Jupyter integration support

[xbwwj]

Describe the enhancement requested

Deno has officially supported Jupyter, and jupyter output can be customized by defining a [Symbol.for("Jupyter.display")] method in the class.

For example, the following code displays a dataframe.

{
  [Symbol.for("Jupyter.display")]() {
    return {
      // HTML output
      "text/html": `
        <table border="1" class="dataframe">
          <thead>
            <tr>
              <th>name</th>
              <th>age</th>
            </tr>
          </thead>
          <tbody>
            <tr>
              <td>Alice</td>
              <td>22</td>
            </tr>
            <tr>
              <td>Bob</td>
              <td>24</td>
            </tr>
          </tbody>
        </table>`,
    }
}

The output is like:

name	age
Alice	22
Bob	24

This can be helpful for interactive data exploration.

Are you interested in having such integration in your project? I can help make a pull request.

[kou]

Could you explain what should we do for Jupyter integration?

[xbwwj]

A method named with symbol Symbol.for("Jupyter.display") is the entry for jupyter integration.

So it may be implemented like below

class RecordBatch {
   // other methods and proerties

  // a method providing jupyter representation
  [Symbol.for("Jupyter.display")]() {
    // different supported mime types, these two is sufficient for most case
    "text/plain": ...,
    "text/html": ...,
  }
}

[kou]

Ah, I understand.

Is there any drawbacks by the Jupyter integration? For example, does it introduce any incompatibility in the current supported platforms?

[xbwwj]

The feature itself does not add any runtime overhead and incompatibility.

However, Symbol and Symbol.for is only supported in ES6 and above, I'm not sure if there is a polyfill for that.

[kou]

FYI: Here are the current supported platforms: https://github.com/apache/arrow-js?tab=readme-ov-file#packaging

[pitrou]

@AntoinePrv You might be interested in this.

[AntoinePrv]

Hello @xbwwj,

I am currently working on Arbalister (very early software).
The idea is to easily visualize dataframe files (CSV, Parquet, Avro...), including very large ones, via an Arrow connector.

What I have also been thinking about, is that we can hook extensions by MIME type. With this, any object in any programming language that define the magic Jupyter visualization method (e.g. _repr_mimebundle_) will have this default representation (this is how objects can render as html for instance).
We could do this for application/vnd.apache.arrow.file.

[xbwwj]

@AntoinePrv Good idea.

I do not have enough bandwidth to work on this issue until next month. Feel free to take over this pull request if you want to improve it.

Also another question: How can Table be represented in this mime type? As far as I know, Table does not natively exist in arrow file, while arrow-js uses Table as its primary API.

[AntoinePrv]

Now that I understand a bit more the setting, I guess there are two different things.

Symbol.for("Jupyter.display") that is how the Deno kernel knows what output to give to Jupyter to display.
If we want some automatic render we'd have to fill something in for Table in this library.
I'm not sure if other Javascript kernel (e.g. xeus-javascript cc @DerThorsten) use the same method name.

Now what this function should return?

• Some HTML data (text/html): works out of the box but we'd have to maintain a function doing this;
• Some IPC data (application/vnd.apache.arrow.stream): Almost free to implement here, but this MIME type is not supported by Jupyter out of the box, so it would require the user to install a Jupyter extension that does not exist today (AFAIK).

Everything I mentioned related to actual files may use similar technique but is not directly related to the solution here.

My 2c: there could be a basic HTML table here that would work out of the box; and later, an application/vnd.apache.arrow.stream (activated by some option?) for when there is a well established extension that offer something more than a basic table. But I am not a maintainer or contributor here.