====== Displaying an External Data Source ======

Blocks version 5 introduces support for external collection databases. This can be very useful for places like museums, often using such databases to keep track of descriptions, metadata, images and other content related to their artifacts and exhibits. Here's Hans Ruedisueli of Bureau Jorwert in The Netherlands – an expert in database-driven presentations – talking about how he uses this feature 5 to integrate external collection databases in presentations made with Blocks. 

{{vimeo>528378235}}

This application note provides an example of such a solution, showing a number of collections based on openly available data and images published by the [[https://maas.museum|Museum of Applied Arts & Sciences]] in Sydney, Australia. They provide an [[https://maas.museum/api-documentation/overview/graphql/|open API]] for accessing their collection database from anywhere in the world. More commonly, such a collection database is intended for internal access only, although some are open to the public, like this one. 

As long as your Block server can reach the database, and it contains the information, images, video and other resources you need, you should be able to integrate its content into your Blocks-based presentations and interactive experiences using the method described in this application note.

The ability to display such external collection data in Blocks is based on two features in particular:

  - A "feed script", which is responsible for communicating with the external database. It selects what data to fetch, and processes the data returned in a way that makes it usable in Blocks.

  - The concept of "child replication", allowing block lists such as the Slideshow, Book, Scroller and the Grid block, to derive their child blocks from the list of objects provided by the feed script, rather than created manually in Blocks. 

Watch this video to see some examples, and a brief overview of how to connect Blocks to an external collection database.

{{vimeo>508779815}}

===== Installation =====

To try out this on your own, start with the [[blocks:app-note:start|general setup]] guide to recreate this application using this {{:blocks:app-note:collection-feed:collectionfeed.zip|Blocks root}}.

  - Add a separate browser window (or a PIXILAB Player) pointed to your Blocks server. 
  - Assign it to the display spot shown in the list.
  - The grid with thumbnail images appears on the display spot.
  - Click a Matchbox car thumbnail to show its full data and additional images.
  - Add a second spot, with a Spot Parameter named //MaasCollection// set to //Stewart// to show steam engines instead, as seen in the video linked above.


===== Anatomy of a Feed Script =====

A feed script has many similarities with a network device driver in Blocks, and requires the same kind of [[blocks:drivers:tools|tools]] and [[blocks:drivers:concepts|skills]]. Thus, you may want to start by learning about how to develop device drivers and user scripts. The following description assumes such familiarity.

To follow along with the descriptions below, open the script/feed/Maas.ts script included in this application note using a suitable  [[blocks:drivers:tools|code editor]].

==== Location and Name of a Feed Script ====

A feed script must be placed under script/feed in your Blocks root directory. It must contain an exported class with the same name as the file (minus its .ts extension). That class must extend the abstract Feed class, found in the Feed API (see script/system_lib/Feed.ts).

==== Establishing Feed Collections ====

A single feed script can make several collections available for use in Blocks, assuming they all share the same general structure and database backend. This is done using the establishFeed method of the Feed base class. In the enclosed example, this establishes a number of collections, giving them appropriate names that will then be used in Blocks to show that collection. You must establish at least one such collection in your feed script.

==== Definition of a Feed Collection ====

Each feed collection established through establishFeed must implement the StaticFeed interface from the Feed API. In doing so, it must provide the following fields and methods:

<code>
name: string;   // Internal (brief) name of this feed instance

listType: Ctor<ListItem>;    // Specifies type of items returned by getList
itemType: Ctor<DetailsItem>; // Type of items returned by getDetails

getList(spec: FeedListSpec): Promise<ListData<ListItem>>;
getDetails(spec: FeedDetailsSpec): Promise<DetailsItem|undefined>;
</code>

The //listType// and //itemType// specifies the type of objects that will be provided by the feed, where itemType must be a superset of listType. Typically, objects of listType contain a small subset of fields – such as just a title and a thumbnail image URL – required for providing an overview of a large number of objects (such as the grid view showing the cars in this example).

==== Obtaining a List of Objects ====

The //getList// method must return a list of objects of listType. This method will be called by Blocks to obtain those objects when used for child replication. Make sure to include all fields needed to show a list of objects. Fields included in objects returned by //getList// appear underlined on the menu used to bind properties in Blocks, letting the Blocks producer know what's available for such overview purposes.

Note that getList returns a //[[https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises|promise]]// for such a list. This allows for the feed script to asynchronously fetch the data from the external database, providing the actual data once it has been obtained. The example included with this application note uses the SimpleHTTP API to fetch the data from the external database, interpreting the data returned (here being JSON data), and passing it on to the //processList// function to re-package the list of objects returned from the database to a list of the type specified by the //listType// field. 

In the included example, the type of objects returned for lists is defined by the //ListItem// class. Here you use @field annotations to define the data fields to be made available to Blocks. These fields will then appear as bindable data, along with the description you provide in the annotations, making it easy to select the fields to show in Blocks.

=== Talking to an External Database ===

Interfacing with an external database can sometimes be rather complicated. The backend used by this example uses [[https://graphql.org|GraphQL]], which provides a great deal of flexibility in querying for only the desired data. The result is returned as JSON, which can be automatically interpreted by Blocks. This automatic interpretation is triggered by the //interpretResponse: true// option passed to newRequest, and requires that the [[https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type|Content-Type]] header of the returned data is set correctly. The resulting data can then be directly used by the feed script. In a similar way, Blocks can auto-interpret XML data. Alternatively, do not pass the //interpretResponse: true// option, making Blocks instead return the raw text of the result, allowing you to parse and interpret it manually in your feed script.

You're not limited to using the SimpleHTTP API for fetching the data (although it's the most common method). You can use any suitable Blocks API for fetching the data, such as SimpleSQL or SimpleFile.

While the data returned from the database hopefully includes the data you need, it's typically not formatted or packaged in the desired way. Thus, your script must re-format the data returned
from the backend into the types to be provided to Blocks, as specified by //listType// and //itemType//. The processList and processDetails functions provide an example of how such data adaptation can be done. The processDetails function, for example, accepts the data returned from the backend call (here described by the //GraphQlSingleItem// interface), and re-packages it into a DetailsItem, expected by Blocks.

==== Obtaining Object Details =====

Once a visitor selects an object from the list to view more details, those details are requested through a call to your //getDetails// method. This typically makes a new round-trip to the backend database, fetching full details for the requested object. Such a request is often based on a unique ID associated with each object in the backend database. Such an ID is annotated with @id in the listType. If no field marked @id is available, you can instead use the index position to fetch the data. This is the 0-based index of the list data returned by getList.

In the included example, the //DetailsItem// class defines whats returned here. Note that DetailsItem extends the ListItem class, and therefore must provide its data as well (i.e., it's a superset of the data provided for each list item).

The DetailsItem can provide additional data fields, such as the "detailed description" field in the example. Such data may be more extensive, such as long textual descriptions, generally not required for an overview list. This two-tiered approach reduces the amount of data fetched initially, loading only the bare minimum for a list overview, then loading full details on demand as individual objects are requested.

==== Providing Nested Lists of Data =====

The getDetails method may also provide nested lists of additional data, annotated with @list, as seen in the enclosed example. This defines a custom data type (here the //Image// class), and returns a list of zero or more such objects. That custom type provides its own @field annotations, describing the data provided by the nested list objects.

Such nested lists can then be used for further child replication, as exemplified by the slideshow of additional images shown in the details view.