Practical Wolf

Recovering Dynkin’s interviews

2020-05-17T00:00:00+00:00

Mathematician Eugene B. Dynkin (1924-2014) conducted over 200 interviews with various mathematicians, and Cornell University Library has digitized this collection. However, that site is currently is broken as it required Flash to play audio and video files, and most of the pages throw PHP errors. In this blog post, we scrap audio and video links to underlying files, along with the metadata, to make this information available again until Cornell fixes its archive.

Extracting interview video/audio links and metadata

The interviews TOC has links to all individual interview pages /node/xxx so that we can get all 200+ of them:

Looking at the individual page’s XML soup, we can see that it has several div pairs with classes field-label and field-items.

In Interview URL the flash player information is URL encoded in an XML element param with name=flashvars, so we need to extract ‘url’: ‘xxx’ parts.

In Translation or Transcription, there are typically a few hyperlinks, so we need to extract them.

For all the other fields, we extract all the strings and concatenate them together.

Parsing the whole collection

Since we might want to revise our extraction logic as we encounter more edge cases, it would make sense to download and parse every URL just once and cache the resulting XML objects.

Here is a full list of fields in the interview metadata, although many of these are missing.

The last touch will be to save the above collection in some sensible format. The easiest would probably be a static HTML file with direct links to Cornell collection. The easiest would be to generate a markdown file with a table that can be converted to HTML by Jekyll that runs this site. The final table is available here.

Download this notebook

A Tale of Two Coasts

2020-04-08T00:00:00+00:00

US West Coast became infected by the coronavirus earlier in the epidemic, with the state of Washington having a first officially recorded COVID-19 case on January 19. Many West Coast companies started WFH policies well ahead of the curve, and a few counties in the Bay Area and Seattle issued “shelter in place” orders before the State did. On the other coast, a mayor of the biggest US city urged his fellow citizens to “go on with your lives + get out on the town despite Coronavirus… thru Thurs 3/5 go see “The Traitor” .”

Let’s compare the dynamics of confirmed cases and growth rates between different US states. While many confounding factors, such as population densities, use of public transportation, use of face masks, weather, administrative restrictions, etc. makes it harder to draw clear conclusions, there is a growing consensus that New York situation got substantially worse because of the city’s delayed response.

Data preparation and helper functions

Note: This section is quite technical; if you just want to look at the results of the analysis, you can skip it.

Before we dive into the analysis, let’s gather the data and define a few functions to make the plotting easier. A more detailed description of the data sources and how to use datasets and Entity is provided in two previous posts here and here.

First, let’s combine the information on population and population densities with COVID-19 confirmed cases and deaths. Merge, as it name implies, is used to join two associations by the same key, and the second argument, a pure function that replaces a head of the list with Join, allows us to merge the nested association together, since both of the data structure that we are trying to merge as associations of associations. (If this is unclear, evaluate each of the arguments to Merge separately, inspect and compare it to the result). We also use KeyMap to convert association keys from Entity to String.

Here is the structure of the result:

We do the same for the list of countries; the only difference is that the COVID-19 country dataset splits the data into administrative regions for US, China, Canada, Australia and UK, so we need to combine them to get the total:

Let’s define a function that would normalize timeseries in the following way: first, it will convert the absolute number of cases or deaths into cases per 10K (or 1M) people, to make the numbers between different states and countries comparable. And second, it will truncate all the values below a certain threshold, so that when we compare the growth over time, all the curves start (roughly) in the same place.

For example, here is the entry for California:

and here is the deaths per million since the day they exceeded 1:

We will also create shortcuts for cases and deaths that can be applied to whole associations:

Finally, let’s define some plotting routines. Since 50 US states (or 251 countries) is much to display on the same graph, we’ll define a function to plot selected lines with colors and labels, while plotting everything else with thin gray lines to create a context.

A Tale of Two Coasts

As we mentioned in the beginning, West Coast states were more proactive than East Coast states in implementing WFH and quarantine measures. Add lower population density and significantly smaller public transportation, and you will see a drastically different picture in terms of both number of confirmed cases:

and number of deaths:

Here is a full grid showing data for each individual state, for both confirmed cases:

and deaths for those cases that have more than 1 per 1M:

Population density and other factors

To put New York numbers in the perspective, let’s compare them to aggregates for a few countries. Let’s see how it compares to Italy and Spain that didn’t do so well, as well as to South Korea - a place with a high population density, as Seoul’s population is 10M people is over 20% of the country’s total.

As you can see, New York substantially dominate all of these countries. South Korea is barely visible at the bottom of the chart, as they have managed the pandemic exceptionally well, with ~4 deaths per 1M. In other observations, overall US numbers track France surprisingly well, and situation is much worse in Spain now that in Italy, which was the original “hot bed” in Europe, with France almost caught to Italy.

Download this notebook

Correlation between COVID-19 per capita death rates and population density for US states

2020-04-05T00:00:00+00:00

We had a Zoom session with a friend yesterday as I was showing him around the Wolfram Language, and one of the questions we discussed is whether there is a correlation between death rate per capita and population density for US states. We could hypothesize that there should be one, In essence, higher population density means higher rates of infection, regardless of what the official number of confirmed cases say, as the testing protocols can be different between different states.

Population density for the continental US states

For a warmup, let’s plot the population densities for the continental US states, which is easy to do as this information is available as one of AdministrativeDivision properties. We can use a convenient EntityValue function, with a proper EntityClass as the first argument, which can be entered by pressing Ctrl+=, typing “us states”, and hitting Enter. You can think of EntityClass as a collection of entities of a particular type.

Since we provided EntityAssociation as a third (optional) argument to EntityValue, the result is an association which maps each US state entity to its population density.

Notice that the population density is not just a simple number, but a number with some magnitude (people per square mile), represented in Wolfram Language by an expression with the Quantity head.

Here, % refers to the result of the previous expression, and FullForm shows how the above expression is stored internally. FrontEnd does a charming job of displaying the entity, quantity, rule, and association nicely.

Having the population density represented as a quantity is very convenient, as we, for example, can convert it into different units:

Before we can display these densities on a heat map, we would need to do a couple of things: first, we want to remove Alaska and Hawaii, and also DC (as it’s a clear outlier). And secondly, we want to apply Log to numerical values, as the difference in densities between a tiny Delaware and a sparsely populated Wyoming is over 60x!

The first task can be done with KeyDropFrom, which drops given keys from association (you can copy and paste the names of the states from the output, or use Ctrl+= again to enter them). The second task can be accomplished by using a combination of QuantityMagnitude, which extracts a numerical value from Quantity, and Log,. Both of this functions are Listable (= can be applied to a list or association without an explicit call to Map), and Map[f, expr] applies f to values when expr is an association. Finally, GeoRegionValuePlot will color the states according to the (log) of their population density:

We can combine the above two steps into a function that, given an entity list of countries or administrative divisions (everything that has a population density property) plots the above. The optional exclude argument can be used to remove certain areas from plotting.

Note that we can also use a collection of countries as an argument to plotPopulationDensities, since each country is an Entity object that also has PopulationDensity property:

Death rates per capita vs population density

Now for the main dish: we’re going to use the same dataset as in our earlier blog post on this topic, and use the fact that AdministrativeDivision column returned by ResourceData is an Entity object:

Let’s take the first line of this dataset as an example (Normal converts a one element dataset into an association):

Let’s write a function that calculates two values - population density and COVID-19 deaths per capita. We also wrap these two values into a callout, so that ListPlot can show these labels:

Now we can apply this function to each row of the dataset (we’ll also remove a couple of outliers) and plot the result:

Visually, there is a weak correlation between log population density and log death rate per capita, as we expected. Let’s calculate the R-square explicitly. We can use Cases to extract just the first argument from Callout. There are two more subtle points: we apply N to callouts, which calculates the numerical value to all the logarithms, and we only select cases where the second value is Real, since for Wyoming, that death per capita is minus infinity.

The R-squared is pretty low, though at only 22%:

Finally, let’s plot the fitted line on the chart:

Download this notebook

Converting Wolfram Notebooks to Markdown

2020-04-02T00:00:00+00:00

We want to convert a Wolfram Notebook into a jekyll markdown to use on this blog. At this point we are only converting 6 type of cells: Section, Subsection, Subsubsection, Text, Input and Output. Here is how to get all of these cells as a list:

Processing text cells

We’ll define a single function, processCell, that will convert cells of a different type into strings. The simplest possible case is a simple text in (sub)section titles or text:

A little bit more complex are cases where there is some formatting inside the text, for example italic, text formatted as input, etc. We’ll define a processTextData function to deal with this:

Processing inputs and outputs

To process inputs and outputs, we need to convert them into a PNG file. We’ll be using two global variables, $imageOutputDir which defines where to output files, and $imageNumber, which is a counter for the current image being output.

Main conversion function

Now that we have defined utilities for converting individual cells, we can define a function to convert a notebook. Since out ultimate goal is do conversion with a press of a single pallette button, we want to define reasonable default values for as many required parameters as possible.

For simplicity, we expect the very first cell to contain a title and be formatted with Section style.

Similarly, the second cell should contain a representation of date.

If neither of this conditions holds, we’ ll signal an error and won’ t continue the conversion. We will also copy the notebook to /assets/notebooks folder, and put the link to the source at the end of the blog post.

Download this notebook

COVID-19 confirmed cases and deaths in the US

2020-03-31T00:00:00+00:00

Wolfram Research maintains and regularly updates COVID-19 data series in their repository and it provides the US state level data.

The data is stored in a Dataset, a Wolfram Language’s way to organize and query structured data. It has 51 entries (for 50 US States and the District of Columbia), and tracks the cumulative number of confirmed cases, recovered cases, and deaths for each state by date.

Cases and deaths are stored as TimeSeries, which is a sophisticated way to represent time-value pairs: for example, you can query the TimeSeries object about the first / last date in the series, ask for a value on a particular date (and it will be automatically interpolated if the data point doesn’t exist), etc.

We can plot the number of cases for New York using DateListLogPlot, which makes the Y-axis logarithmic. We also start plotting from Mar 2nd, as there is no interesting data before:

Dataset has a powerful query operator syntax: to quote from the official documentation,

In dataset[op1, op2, ...], query operators op_i are applied at successively deeper levels of data, but any given one may be applied either while “descending” into the data or while “ascending” out of it.

There are many special cases: for example, a single number i is a descending operator which takes part i and applies subsequent operators to it, while "key" similarly takes the value of key in the association. Therefore usData[1, "ConfirmedCases"] takes the first row of the dataset, which is an association with six keys shown above, and then takes the value of "ConfirmedCases" key, returning a TimeSeries.

Here is another example: since we are not going to use Country and GeoPosition fields, we can redefine usData to omit those fields. In this case, All is an operator that applies subsequent operators to every part of the list or association, and <|key1->op1, key2->op2, ...|> applies multiple operators at once to the result, yielding an association with the given keys, and also let us rename the columns.

While we are here, we will also convert state names to strings, since every state returned by ResourceData is an Entity object, not a string, and a combination of StringSplit and First extract just the first part before the comma. We also use /*, which represents a right composition of functions.

Using a helper function that returns the last value for a time series, we can see the latest number of confirmed cases and deaths. Here we used another query syntax: {key1->op1, key2->op2, ...} applies different operators to specific parts in the result.

We can now calculate and plot daily growth in number of cases and deaths. To avoid having too much noise in the charts, we will only calculate differences once a state has greater than 500 confirmed cases or 20 deaths. Note the use of a helper function growthAfter and how we use pure functions

Similarly, we will only plot a data series if it has more than 5 data points for deaths or 9 data points for cases:

Since we want to plot a few similar charts with different parameters, we’ll use a helper function for that:

Here is a chart with the number of daily increases for deaths - New York is clearly dominating everything else

Here is the same chart without New York: California has a recent spike, while Massachusetts remains pretty stable for a long time.

And here is the number of confirmed cases, with and without NY and NJ scale: Washington, which had the first case in the country, and New Jersey have somewhat stabilized, but it can just be a function of how much testing is done at a state level.

Download this notebook