Creating a Twitter Graph Using Slash GraphQL
Continuing my personal journey into learning more about Dgraph Slash GraphQL, I wanted to create a graph visualization of data stored in a graph database. Graph visualization (or link analysis) presents data as a network of entities that are classified as nodes and links. To illustrate, consider this very simple network diagram:
While not a perfect example, one can understand the relationships between various services (nodes) and their inner-connectivity (links). This means the X service relies on the Y service to meet the needs of the business. However, what most may not realize is the additional dependency of the Z service, which is easily recognized by this illustration.
For this article, I wanted to build a solution that can dynamically create a graph visualization. Taking this approach, I will be able to simply alter the input source to retrieve an entirely different set of graph data to process and analyze.
Instead of mocking up data in a Spring Boot application (as noted in the “Connecting Angular to the Spring Boot and Slash GraphQL Recommendations Engine” and “Tracking the Worst Sci-Fi Movies With Angular and Slash GraphQL” articles), I set a goal to utilize actual data for this article.
From my research, I concluded that the key to a building graph visualization is to have a data set that contains various relationships. Relationships that are not predictable and driven by uncontrolled sources. The first data source that came to mind was Twitter.
After retrieving data using the Twitter API, the JSON-based data would be loaded into a Dgraph Slash GraphQL database using a somewhat simple Python program and a schema that represents the tweets and users captured by
twarc and uploaded into Slash GraphQL. Using the Angular CLI and the ngx-graph graph visualization library, the resulting data will be graphed to visually represent the nodes and links related to the #NeilPeart hashtag. The illustration below summarizes my approach:
Retrieving Data From Twitter Using `twarc`
While I have maintained a semi-active Twitter account (@johnjvester) for almost nine years, I visited the Twitter Developer Portal to create a project called “Dgraph” and an application called “DgraphIntegration”. This step was necessary in order to make API calls against the Twitter service.
The twarc solution (by DocNow) allows for Twitter data to be retrieved from the Twitter API and returned in an easy-to-use, line-oriented, JSON format. The twarc command line tool was written and designed to work with the Python programming language and is easily configured using the
twarc configure command and supplying the following credential values from the “DgraphIntegration” application:
With the death of percussionist/lyricist Neil Peart, I performed a search for hashtags that continue to reference this wonderfully-departed soul. The following search command was utilized with twarc:
Below is one example of the thousands of search results that were retrieved via the twarc search:
Preparing Dgraph Slash GraphQL
Starting in September 2020, Dgraph has offered a fully managed backend service, called Slash GraphQL. Along with a hosted graph database instance, there is also a RESTful interface. This functionality, combined with 10,000 free credits for API use, provides the perfect target data store for the #NeilPeart data that I wish to graph.
The first step was to create a new backend instance, which I called
Next, I created a simple schema for the data I wanted to graph:
Tweet types house all of the data displayed in the JSON example above. The
Configuration type will be used by the Angular client to display the search string utilized for the graph data.
Loading Data into Slash GraphQL Using Python
Two Python programs will be utilized to process the JSON data extracted from Twitter using twarc:
- convert – processes the JSON data to identify any Twitter mentions to another user
- upload – prepares and performs the upload of JSON data into Slash GraphQL
The core logic for this example lives in the upload program, which executes the following base code:
gather_tweets_by_user()organizes the Twitter data into the
upload_to_slash(create_configuration_query(search_string))stores the search that was performed into Slash GraphQL for use by the Angular client
forloop processes the
userobjects, uploading each record into Slash GraphQL using
Once the program finishes, you can execute the following query from the API Explorer in Slash GraphQL:
Using `ngx-graph` With Angular CLI
The Angular CLI was used to create a simple Angular application. In fact, the base component will be expanded for use by
ngx-graph, which was installed using the following command:
npm install @swimlane/ngx-graph --save
Here is the working AppModule for the application:
In order to access data from Slash GraphQL, the following method was added to the GraphQlService in Angular:
Preparing Slash GraphQL to Work With `ngx-graph`
The data in Slash GraphQL must be modified in order to work with the
ngx-graph framework. As a result, a ConversionService was added to the Angular client, which performed the following tasks:
The resulting structure contains the following object hierarchy:
While this work could have been completed as part of the load into Slash GraphQL, I wanted to keep the original source data in a format that could be used by other processes and not be proprietary to
Configuring the Angular View
When the Angular client starts, the following
OnInit method will fire, which will show a spinner while the data is processing. Then, it will display the graphical representation of the data once Slash GraphQL has provided the data and the ConversionService has finished processing the data:
On the template side, the following
ngx tags were employed:
ng-template tags not only provide a richer presentation of the data but also introduce the ability to click on a given node and see the original tweet in a new browser window.
Running the Angular Client
With the Angular client running, you can retrieve the data from the Slash GraphQL by navigating to the application. You will then see a user experience similar to below:
It is possible to zoom into this view and even rearrange the nodes to better comprehend the result set.
Please note: For those who are not fond of the “dagre” layout, you can adjust the
ngx-graph.layout property to another graph layout option in
When the end-user clicks a given node, the original message in Twitter displays in a new browser window:
A fully-functional Twitter Graph was created using the following frameworks and services:
Twitter API and Developer Application
twarc and custom Python code
Dgraph Slash GraphQL
Angular CLI and
In a matter of steps, you can analyze Twitter search results graphically, which will likely expose links and nodes that are not apparent through any other data analysis efforts. This is similar to the network example in the introduction of this article that exposed a dependency on the Z service.
If you are interested in the full source code for the Angular application, including the Python import programs referenced above, please visit the following repository on GitLab:
Have a really great day!
Credit: Source link