FactMiners.org - AI etc.

Inside the FactMiners' Brain - Rainman Meet Sherlock

Jim Salmons — Sun, 31 Aug 2014 17:27:22 +0000

NOTE: In case you missed it, here is a link to a screencast of Kenny Bastani's webinar about using the Neo4j graph database in text classification and related Deep Learning applications. It's a fascinating introduction to some original work Kenny is doing that leverages the strengths of a property graph, in this case Neo4j, to do some Deep Learning text-mining and document classification.

This article is about what we are going to try to do with Kenny's new Graphify extension for Neo4j. And a big "Thank you!" and kudos to Kenny for kickstarting activity around this important topic within the Neo4j community.

Some Thoughts About Thinking

You would be hard-pressed to go through any formal education where you did not learn about our "two brains" – left and right hemispheres, verbal/non-verbal, creative/literal, the conscious/subconscious, long-term/short-term, self/other, etc. All these various perspectives remind us that how we think, as humans, is a complex yin-yang cognitive process. Whatever works well to help us understand ourselves and the world around us in some cases, does poorly helping us in others, and vice versa. So we've cleverly evolved the "wetware" to do both and, in one of our brains' most truly amazing feats, to provide some kind of highly effective, real-time integration of the these multiple perspectives.

One of the most intriguing distinctions to consider when attempting to model human cognitive processes (let's settle for calling it "smart software" to avoid going too far into pure ResearchSpeak) is to look at the role of subconscious and conscious processing. Some things are either so voluminous and detailed -- basic perception, for example -- that we would bore ourselves to death and slow our thinking processes to a crawl if they ran through our conscious, mostly verbal, cognitive processes. Some other aspects of our thinking -- e.g. things that produce an "A-ha!" conscious moment of discovery -- require the "hands off" focus of subconscious processing. Without such cloistered incubation opportunities, our overbearing conscious mental processes can too easily derail an otherwise breakthrough thought.

So it should not surprise us that software analogs of (something akin to) our own cognitive processing will benefit from providing a similar strategy of "complementary opposites." We should expect to find some real design opportunities for "smart software" by providing a rough approximation of this subconscious/conscious distinction as we move from an application-centric development mindset to a more appropriate agent-centric design mindset. Exploring how smart software might incorporate this "two-cyclinder thinking engine" is one of the "serious fun" R&D initiatives at FactMiners.org.

We're active in the Neo4j community because FactMiners is exploring the unique, expressive nature of graph database technology to model both how "subconscious" cognitive processing (e.g. the NLP-based stuff of Kenny's text classification webinar) can be integrated with "conscious" cognitive processing (e.g. our metamodel-subgraph GraphGists that are more akin to "mind maps"). Our belief is that such a software design strategy can lead to a synergistic result that is greater than the sum of what these simulated cognitive processes can contribute independently. To allude to popular culture, our research asks:

How can we get both the Rainman-like, obsessive-compulsive, bureaucratic, ruthlessly-detailed part of our subconscious processing to work in concert with the Sherlock Holmes-like, logical, deductive, constructive part of our "wetware"?

The Rainman Part - Kenny Bastani's Text Classification Blog/Webinar

Kenny Bastani's webinar this Thursday, "Using Neo4j for Document Classification" will provide a great live demonstration of the kind of relentless, detail-oriented, largely subconscious aspect of our human cognitive process. Kenny's recent blog post, "Using a Graph Database for Deep Learning Text Classification," is provided as a webinar supplement and gives a good introduction (with links) to the Deep Learning ideas and methods employed in his latest Open Source project, Graphify.

Graphify is a Neo4j unmanaged extension adding NLP-based (Natural Language Processing) document and text classification features to the Neo4j graph database using graph-based hierarchical pattern recognition. As Kenny describes in his blog post:

"Graphify gives you a mechanism to train natural language parsing models that extract features of a text using deep learning. When training a model to recognize the meaning of a text, you can send an article of text with a provided set of labels that describe the nature of the text. Over time the natural language parsing model in Neo4j will grow to identify those features that optimally disambiguate a text to a set of classes." (Kenny Bastani, full post)

When you read the rest of Kenny's blog post you will get a very quick and informative introduction to the Vector Space Model for Deep Learning representation and analysis of text documents. The algebraic model underlying Kenny's Graphify Neo4j extension is just the kind of Rainman-like, obsessive, detail-oriented processing that is representative of the subconscious side of our cognitive processing.

If you read the above description of Graphify closely, you will see the opportunity for synergy and integration between Graphify's "subconscious" processing and the more "conscious" processing reflected in my GraphGists exploring the "self-descriptive" Neo4j graph database.

The Sherlock Part - FactMiners' Metamodel Subgraph GraphGists

Imagine if you were able to sit down for tea with the fictional Sherlock Holmes. We hand him a paper and pen, and then ask for a description of the particulars of his latest case. Sherlock would surely resort to sketches in the form of a graph diagram or something easily mapable to a graph representation. Graph semantics are "elementary" and flexibly extensible -- properties that Sherlock would surely appreciate.

I started exploring this "conscious" cognitive process side of graph database application design in the first two parts of my GraphGist design document series, "The 'Self-Descriptive' Neo4j Graph Database: Metamodel Subgraphs in the FactMiners Social-Game Ecosystem." In the longer and more detailed second part of this GraphGist, I explored how an embedded metamodel subgraph can be used to model a "Fact Cloud" of Linked Open Data to be mined from the text and image data in the complex document structure of a magazine. In our case, we'll use FactMiners social-gameplay to "fact-mine" a digital archive of the historic Softalk magazine which chronicled the early days of the microcomputer revolution. In this regard, our "sandbox-specific" application is museum informatics. However, there is nothing domain-specific about the solution design we are pursuing.

With this more general application in mind and in looking for that opportunity where Sherlock can work hand-in-hand with Rainman, it is the first part of this GraphGist series that is the more relevant to the "whole brain" focus of this post.

In this first part of my GraphGist, I provide a "Hello, World!" scale example of how a graph database can be 'self-descriptive' to a layer of smart software that can take advantage of this self-descriptive nature. In this gist, I had some fun exploring the old aphorism from Journalism school, "Dog bites man is nothing, but man bites dog, that's news!"

In brief, the assumption is that a 'self-descriptive' database is 'talking' to something that is listening. Under this design pattern, the listening is done by a complementary layer of "smart software" that can use this information to configure itself for all manner of data analysis, editing, and visualization, etc.

In the case of the ultra-simple "Man bites Dog" example, the layer of smart software is nothing more elaborate than some generalized metamodel-aware Cypher queries. Cypher is the built-in query language for Neo4j. In my gist example, these queries are used for "news item" discovery and validation. By simple extrapolation you can readily imagine the level of "conscious" processing that could be brought to bear to "think about" the data in a 'self-descriptive' graph database.

With this much overview of the "subconscious" and "conscious" aspects of our FactMiners' brain, we're ready to look at that opportunity for integration... that place where Rainman meets and works with Sherlock.

How Rainman and Sherlock Might Work Together

There is a strong hint at how the Deep Learning "subconscious" processing of Kenny's Graphify component might fit into the "brain model" of FactMiners. I'll underline the bits in this quote from Kenny's article to suggest the integration opportunity: "Graphify gives you a mechanism to train natural language parsing models that extract features of a text using deep learning. When training a model to recognize the meaning of a text, you can send an article of text with a provided set of labels that describe the nature of the text..."

Graphify just needs to be fed a list of those "text nature describing" labels and a pile of text to dive into and away it goes. I believe an excellent source of this "text nature knowledge" -- those labels needed to seed the training and data extraction of Kenny's "subconscious" Rainman-like text classification process -- those labels are very explicitly represented, maintained and extended in the information encoded in, and organized by, the metamodel subgraph of a 'self-descriptive' graph database.

We should be able to establish a feedback loop where Graphify's label list is supplied by Sherlock's "mental model" in the metamodel subgraph, and Graphify's results are fed back to refine or extend the metamodel. How, even whether, this will all work as envisioned we will discover over the weeks ahead.

Next up? We're looking forward to Kenny's webinar and to having some serious fun digging into Graphify.

Karma to Take a LOD off FactMiners

Jim Salmons — Wed, 07 May 2014 21:36:22 +0000

One of the beauties of doing a grassroots Open Source community project is that we are not just open in the terms of our licensing, etc., but open to collaboration and incorporation through sharing both ideas and code. This is why we were ecstatic to learn about Karma.

Karma is an amazing Open Source "multilingual" ontology-aware cross-model smart-mapper providing "Rosetta Stone"-like powers to users coping with the ever-shifting publication of Linked Open Data (LOD). Karma is the evolving brilliant work from the incredible team of researcher-makers led by Craig Knoblock and Pedro Szekely of the Information Sciences Institute at the University of Southern California.

If we are lucky – because this would be an incredibly difficult, if not impossible, piece of work to duplicate as a subproject within FactMiners itself – Karma will handle this critical "LOD-switchboard" service as a component of the technology stack for the FactMiners social-game ecosystem. (The Karma codebase is surely going to inform the design of, if not directly contribute as an included component to, the FactMiners Fact Cloud Wizard.)

I could go on and on about the incredible work the Karma folks have done, but as seeing is believing, please watch and marvel at this demonstration:

Note: Karma is envisioned and implemented as a much more general data integration tool than the specific use case that excites us. This is an incredible resource for anyone needing "Rosetta Stone"-like features for both data integration and new model generation. All that said, we're incredibly thankful that they are doing such a great service to the exploding Linked Open Data (LOD) World of which cultural heritage repositories of Libraries, Archives, and Museums (LAM) are among the most "explosive." This exploding LAM/LOD World is also a virtually limitless expanse of prospective FactMiners' playgrounds.

Why FactMiners Loves Karma

A core idea of the FactMiners social-gaming platform is that the FactMiners gameplay produces a by-product which is its collection of Fact Clouds. At their core – and we skim momentarily into just-enough implementation territory – FactMiners Fact Clouds are Neo4j graph databases with an embedded metamodel subgraph providing a rich and extensible "self-descriptive" resource that describes and validates the "actual" data in the Fact Cloud database. In other words, we have a very "civilized" inner-world where each Fact Cloud will be a remarkably expressive datastore of semantically-rich, queriable, discoverable, analytically viewable "facts."

No matter how organized and accessible our Fact Clouds might be internally, responsibly putting these new data resources on-line through LOD publication is itself a daunting and ever-evolving challenge. LOD publishers – which FactMiners Fact Cloud creators will very likely be – need a "smart switchboard" to handle the "semantic pipes" feeding in and out of their semantically-rich datastores, that is, FactMiners Fact Clouds in our case. This daunting task is the exact thing that these incredible research scientists have tackled in birthing and nurturing Karma. The fact that this resource is an Open Source project rather than an enterprise-scaled and priced not-for-you technology is remarkable and greatly appreciated.

We'll keep you abreast of our progress exploring the remarkable resource of Karma. In closing, I'd like to thank Robert Stein, Deputy Director of the Dallas Museum of Art and a Director of the Museum Computer Network, for the referral and introduction to Pedro Szekely. I am looking forward to a planned chat with Pedro after the current USC semester settles into a dull roar.

So, bottom line, Karma is awesome. Hey, www.Structr.org devs... you are SO going to like what this can do for the FactMiners/Structr platform! :D --Jim--

The Pursuit of Serious Fun with Images and Robots

Jim Salmons — Fri, 10 Jan 2014 22:24:39 +0000

Can kids and parents – students and tutors – have fun learning and teaching together AND create something that will contribute to advancing the state-of-the-art of computer vision and artificial intelligence research?

In 2009 when Dr. Li and kindred computer vision and artificial intelligence (CV/AI) researchers looked to the Internet for real world data to test their machine-learning programs doing full scene image recognition, user-tagged collections of images on sites like Flickr were about as rich a learning resource as could be found. Dealing with "dirty"/irrelevant tags in such image collections is a non-trivial challenge for these CV researchers. And it is certainly reasonable for the study designs of these researchers to have assumed scarcity of (assumed-expensive) human resources for both materials prep and interactive tutoring/training of machine-learning programs.

By 2015 the 'Seeing Eye Child' Robot Adoption Agency plans to have both:

a semantically-rich Fact Cloud for a non-trivial subset of the British Library Image Collection, AND
a game-energized, crowdsource-powered human-tutor resource freely available as a CV/AI machine-learning program training resource

Through initial counsel and collaboration with active CV/AI researchers, we will refine and extend our game design and community dynamics to transition the 'Seeing Eye Child' Robot Adoption Agency into its mature and sustainable state.

At sustainable maturity, the Robot Adoption Agency gaming community will attract programming learner-players who will use the game's "sandbox" resource and community to develop and extend their CV/AI skills and interests. Some proportion of those programmer-players will develop a deep interest in human/computer interaction and contribute to the Robot Adoption Agency's gaming community by creating such components as new Open Source training/tutor workflow plug-ins. Those with interests driven more by game design and development will likely contribute presentation/interaction plug-ins to add fun and engaging robot character generators for our programmer-players' otherwise unseen running-in-memory agent programs. When we get to this level of community self-support, the game will be in its own good hands.

So, what's next? Will the FactMiners ecosystem ever be more than just an interesting idea that remains untried? I, for one, don't intend to let that happen. Step by step, we're moving forward. In the "FactMiners: More or Less Folksonomy?" article, we have reached out and begun collaborations with museum informatics professionals, both for the counsel of their domain expertise and to find kindred spirits interested in hosting FactMiners Fact Cloud companions for their on-line digital collections. In this article, we've described how the FactMiners ecosystem and its Fact Cloud architecture can accommodate image-based digital collections in addition to the print/text realm of complex magazine document structure of our project focus at The Softalk Apple Project.

In exploring this new use case within digital image collections for the FactMiners ecosystem, we have identified how our game design can "play" into the domains of computer vision (CV) and artificial intelligence (AI). So among our next steps along the path of bringing the FactMiners ecosystem to life will be to find some kindred spirits in the CV/AI domain interested in exploring just how fun (and useful) it would be to have a British Library Image Collection Fact Cloud companion and 'Seeing Eye Child' robot-tutor web service.

I believe if we can bring the active interest of a CV/AI collaborator to the table as we discuss this idea further with the good folks at the British Library Labs, we'll be a BIG step closer to opening the Internet's first 'Seeing Eye Child' Robot Adoption Agency courtesy of the collective efforts of the FactMiners developer community, the British Library Labs, and some as-yet-unidentified CV/AI researchers. Stay tuned...

A Quick Trip to the Stanford Vision Lab

Jim Salmons — Wed, 01 Jan 2014 22:11:31 +0000

AI etc.

Fei-Fei Li is the director of the Computer and Human Vision Lab within the legendary Stanford Artificial Intelligence Laboratory. While her research interests and breakthrough contributions to the field are wide-ranging, I want to focus briefly on a 2009 study she and her colleagues did at Princeton, before Dr. Li's selection to head the prestigious Stanford Vision Lab. "Towards Total Scene Understanding: Classification, Annotation and Segmentation in an Automatic Framework" is a remarkable project and representative of the kind of machine-learning image recognition capabilities envisioned at "serious play" in the 'Seeing Eye Child' Robot Adoption Agency game.

Researchers like Dr. Li are creating machine-learning programs that can effectively 'look' at a previously unseen image and not just find objects within the image but interpret the full scene. This example graphic from the project's summary shows how a total scene is considered as a comprehensive model that incorporates both top-down context, e.g., a polo match, along with both visual and textual (tag) elements. For Dr. Li and associates' study, the tags and images were drawn from Flickr – yes, the same popular image-sharing site where the British Library released its 1-million-plus public domain image collection – and the end-to-end automatic machine-learning/scene-recognition process is generally described as shown in the following 3-step workflow diagram from the study.

The results of this machine-learning strategy are both remarkable and encouraging for our game design requirements. Given a 'seed' set of curated images with text label 'hints' and a representative collection of similar-scene images clustered by the eight targeted sports categories, the researchers' automated process does a remarkable job of finding and labeling these elements in new unseen images, producing results such as this test image which has been correctly recognized as a polo scene with horse and rider, trees, and grass:

While Dr. Li and associate's strategy is particularly robust and comprehensive, their paper cites 22 additional studies in four broad areas of CV/AI research that tackle the full scene understanding challenge:

Image understanding using contextual information
Machine translation between words and images
Simultaneous object recognition and segmentation
Learning semantic visual models from Internet data

So it is safe to say that the domains of computer vision and artificial intelligence (CV/AI) are at a sufficient stage of capability and active research that the kind of 'robot' vision required for our game design is both doable and getting better. If you have any doubts (or better yet, interest to know more), I encourage you to read Dr. Li's project overview, or better yet, the full study PDF and follow links to cited related research.

In the concluding post of this series about the potential for FactMiners to contribute to the "serious fun" at the British Library Image Collection, I'll set some goals and chart a course forward to add the 'Seeing Eye Child' Robot Adoption Agency to the selection of social-learning games to be developed by, and available to, the FactMiners gaming community.

Finding the 'CV' in 'STEM' at the British Library Image Collection

Jim Salmons — Sun, 29 Dec 2013 01:33:12 +0000

AI etc.

Computer Vision, or using its popular acronym 'CV', is a domain of scientific knowledge and practice within the domain of Artificial Intelligence which is part of the broad domain of Computer Science. CV is a particularly challenging field that has strong connections into each of the Science, Technology, Engineering, and Math 'branches' of the STEM fields of education.

You only have to glance at the image here of a modern auto assembly line and compare it to one of our not-so-distant mid-20th century lines, or watch the workerbot demo video to know this: What is currently an exciting bleeding edge of CV and AI research and industrial practice today will be mainstream vital skill areas for both entrepreneurial and employment opportunities in the near and foreseeable future.

And how will this growing cohort of budding job creators and job fillers gain their skills in Computer Vision programming? If they get to play the 'Seeing Eye Child' Robot Adoption Agency game during their formative years, a growing number of them could develop strong CV and AI programming skills and interests by writing FactMiners 'Robot players' to be effectively 'put up' for adoption in the game's Adoption Agency. Role-wise, these 'robot' player/programs are the 'real' player's agent-actor programs. To borrow the anthropomorphic imagery of Disney's 1982 scifi classic, TRON, our young programming "Flynn" user/players will be sending their "Clu" agent/programs into the Robot Adoption Agency to begin a cyber-learning journey into the British Library Digital Image Collection.

From an 'ends-testing' perspective our concern for the game's Adoption Agency robot supply can more creatively be seen, not as a robot (i.e. agent-program) supply problem, but rather as a robot-programmer supply opportunity. Our game's desirable side-effect is to act as part of its own positive feedback loop whereby the demand for more and better CV/AI programs in the emerging Internet of Things will generate demand for more and better robot programmers.

Having both means- and ends-tested the motivation and justification for this game design idea, a fundamental question remains... Is machine-learning image recognition like what is assumed to be available in the proposed Robot Adoption Agency game even possible? And if possible, is there a place for human-mediated vision training (the 'Seeing Eye Child' player's role) as envisioned in the game? To consider these important questions, let's take a quick trip to the legendary Stanford AI Lab.

Introducing the 'Seeing Eye Child' Robot Adoption Agency

Jim Salmons — Wed, 25 Dec 2013 22:03:17 +0000

In the first part of this informal proposal to creatively tap the newly-published British Library Image Collection, I imagined a plug-in game to be developed as part of the FactMiners social-game ecosystem. In this adult/child-interactive early-learning app, gameplayers collectively contribute to building a Fact Cloud of "What's in this picture" facts (elementary sentence-like assertions stored in a graph database) for the over one-million images recently uploaded to the Flikr Commons. Parents playing this new word/picture FactMiners plug-in game with their kids create the Fact Cloud that becomes a vital resource for a second new FactMiners game; the 'Seeing Eye Child' Robot Adoption Agency.

The 'Seeing Eye Child' Robot Adoption Agency is similar to the 'Tamagotchi' or 'digital pet' gaming phenomena that hit in the mid-1990's and is still going strong. The difference here is that we harness our little cognitive learning machines – AKA FactMiners game players – to 'adopt' a robot (AKA a machine-learning program with some form of vision – image intake and analysis – capability) and help it to learn to see and understand its world. As an adoptive 'Seeing Eye Child', players take on the roles of coach and referee for training sessions where adopted robots learn to see what's in the British Library images.

The nimble-thinking among you have likely spotted the weak link in this proposed game design... robot supply.

Are we honestly to believe that the latest industrial robots ready to be brought on-line to a Ford or Toyota auto assembly line are in need of vision-training sessions with young kids mentoring their ability to recognize scenes from 17th-19th century book illustrations? Well, that is most certainly not the case.

So how can the Fact Cloud creators – those playing the word/picture FactMiners game that creates the Fact Cloud descriptive companion to the British Library Image Collection – how can these players be motivated to create that Fact Cloud if its imagined great use in robot vision training turns out not to be a need at all? What kid is going to wait around a Robot Adoption Agency match-making server's 'waiting room' for an adoptable robot that may never show up?

Fortunately we can consider both the means and the ends of the Fact Cloud creation effort to answer such important questions. From a 'means value' perspective, the image-describing FactMiners gameplay that creates the Fact Cloud is a fun, social, interactive learning activity. There is an immediate and personal motivation and value for parents, siblings, tutors, and teachers to help little learners build the British Library Image Collection Fact Cloud. So even if the robot vision training need were to turn out to be an elusive future-imagining, the 'serious fun' of building the British Library Image Collection Fact Cloud is time and energy well-spent in direct, interactive childhood and developmental educational activity.

Having 'means-tested' the effort to create the British Library Image Collection Fact Cloud, in my next post I will turn our attention to the 'ends' test – Will we really have a robot supply problem at the 'Seeing Eye Child' Robot Adoption Agency?...

A FactMiners' Fact Cloud for the British Library Image Collection

Jim Salmons — Sun, 15 Dec 2013 21:21:20 +0000

I was thrilled to read the announcement this week in the British Library Digital Scholarship blog about the Library's uploading to the Flickr Commons of over 1 million Public Domain images scanned from 17th, 18th, and 19th century books in the Library's physical collections. The Flickr image collection makes the individual images easily available for public use. Currently the meta-data about each image includes the most basic source information but nothing about the image itself. In the words of project tech lead Ben O'Steen:

We may know which book, volume and page an image was drawn from, but we know nothing about a given image. Consider the image below. The title of the work may suggest the thematic subject matter of any illustrations in the book, but it doesn't suggest how colourful and arresting these images are.

See more from this book: "Historia de las Indias de Nueva-España y islas de Tierra Firme..." (1867)

We plan to launch a crowdsourcing application at the beginning of next year, to help describe what the images portray. Our intention is to use this data to train automated classifiers that will run against the whole of the content. The data from this will be as openly licensed as is sensible (given the nature of crowdsourcing) and the code, as always, will be under an open license.

Ben went on to explain, "Which brings me to the point of this release. We are looking for new, inventive ways to navigate, find and display these 'unseen illustrations'."

Well, Ben's challenge got me thinking... What would be the value of creating a FactMiners' Fact Cloud Companion to the British Libary Public Domain Image Collection?

And that's when I had my latest "Eureka Moment" about why the FactMiners social-game ecosystem is such a compelling idea (at least to me and a few others at this point :-) ). First, let me briefly describe what a Fact Cloud Companion would look like for the British Library Image Collection before exploring why this is such an exciting and potentially important idea.

A FactMiners Fact Cloud for Images: What?

When Ben laments that the Library's image collection does not know anything about the content of the individual images, I believe he 'undersold' that statement by alluding to the metadata not informing us how colorful or arresting this image is. But there is a much more significant truth underlying his statement.

Images are incredible "compressed storage" of all the "facts" (verbal assertions) that we instantly understand when we humans look at an image. The image Ben referenced above of the man in a ceremonial South American tribal regalia is chuck full of "facts" like:

The man is wearing a mask.
The man is wearing a blue tunic.
The man is holding a long, pointed, wavy stick.
The man has a feathered shield in his left hand.
The man is standing on a fringed rug.
The man has a beaded bracelet on his right arm.

I've written briefly about how an Open Source graph database, like Neo4j, is an ideal technology for capturing FactMiners' Fact Clouds. So I won't belabor the point by drilling down here on these example 'image facts' to the level of graph data insertions or related queries. Suffice to say that the means are readily available to design and capture a reasonable and useful graph database of facts/assertions about what is "seen" in the "unseen illustrations" of the British Library image collection.

Rather, I want to move on quickly to the "A-ha Moment" I had about why creating a Fact Cloud Companion to the British Library Image Collection could be a Very Good Thing.

A FactMiners Fact Cloud for Images: Why?

Every time we look at an image, our brains decompress that in an "explosion of facts." By bringing image collections into the FactMiners' "serious play arena" we are, in effect, capturing that "human image decompression" process as a sharable artifact rather than it being a transient individual cognitive event. In other words, every child goes through the learning process of "seeing" what's in a picture. When these "little learning machines" do a proportion of that natural childhood learning activity by playing FactMiners at the British Library Image Collection, we get a truly interesting 'by-product' in the Fact Cloud Companion.

Beyond the obvious use of a Fact Cloud for folksonomy-class applications supporting source collection public and researcher access, a FactMiners Fact Cloud Companion of the British Library Public Domain Image Collection would be an invaluable resource for that new emerging museum and archive visitor base... robots. Well, not so much the fully anthropomorphized walking/talking robots, at least not so much just yet. I'm thinking here more like machine-learning programs, specifically those with any form of 'image vision' capability – whether by crude file/data 'input' or real-time vision sensors.

Upon entering the British Library Image Collection, our robot/machine-learning-program visitors would find a rich 'playground' in which to hone their vision capabilities. All those Fact Cloud 'facts' about what is 'seen' in the collection's previously 'unseen images' would be available at machine-thinking/learning speed to answer the litany of questions – "What's that?", "Is that a snake?", "Is that boy under the table?" – questions that a machine-learning program might use to refine its vision capabilities.

So while the primary intent of the project is making these images available for Open Culture sharing and use, there may be some equally valuable side effects of this project. The British Library Image Collection and its Fact Cloud Companion could become a "go-to" stop for any vision-capable robot or machine-learning program that aspires to better understand the world it sees.

A FactMiners Fact Cloud for Images: How?

As the good folks at the British Library well know, just getting a good folksonomy social-tagging resource developed for such a huge collection is itself no small task. This is why museums and archives, like the British Library and those collaborating in the steve project, are turning to crowdsourcing methods to get the 'heavy-lifting' of these tasks done. Crowdsourcing goes hand-in-hand with gamification in this regard. If we can't pay you to help us out, at least we can make the work fun, right?

Well, you don't have to think too hard to realize that if creating a folksonomy is a big chore, then creating a useful Fact Cloud representing at least a good chunk of the 'seen' in the previously 'unseen illustrations' of the British Library Image Collection is a Way Too Big Chore. And this might be true. But I think that there is some uniquely wonderful 'harness-able labor' to be tapped in this regard.

I know we can make a really fun app where parents and older folks can help kids learn by playing; building fact-by-fact a valuable resource at the British Library, for one. A learning child is a torrent of cognitive processing. Let a stream of that raw learning energy run through the FactMiners game at the British Library Image Collection and you'd have critical mass in a Fact Cloud faster than you can say, "Danger, Will Robinson!"

And where might this lead? Well, where this all might lead Big Picture wise is beyond the scope of this post. But I can see it leading to a new, previously unimagined game to add to the mix of social games available to FactMiners players... and it's a bit of a doozy. :-)

If the British Library creates a FactMiners Fact Cloud Companion to its Image Collection, and if that Fact Cloud becomes useful to robots (machine-learning programs) as a vision-learning resource, I can see where we would want to add a 'Seeing Eye Child' Robot Adoption Agency Game to the FactMiners game plug-ins. What would that game be like?

Well, as good as an Image Collection Fact Cloud might be to learn from, and as smart as a machine-learning program might be as a learner, a robot's learning to see isn't likely to be a fully automated process. So we create a game where one or more kids 'adopt' a robot/machine-learning program to help it learn. In this case, the FactMiners player would gain experience points, badges, etc. by being available for 'vision training' sessions with the adopted robot. The FactMiners player is, in effect, the referee and coach to the robot as it learns to see.

It doesn't take much imagination to see how this could lead to schools fielding teams in contests to take a 'stock' robot/machine-learning-program and train it to enter various vision recognition challenges. And when I let my imagination run with these ideas, it gets very interesting real fast. But any run, even of one's imagination, starts with a first step.

Will we get a chance to make a Fact Cloud Companion to the British Library Image Collection? I don't know. This week the British Library took a million first steps toward making their vast digital image collection available to all for free. Perhaps the first step of posting this article will lead us on a path where we will have some serious fun working with the Library to help kids who help robots learn to see and understand our world.

--Jim Salmons--
Cedar Rapids, Iowa USA

Update: An encouraging reply of exploratory interest from the good folks at the British Library Labs has juiced my motivation to further explore the potential for the 'Seeing Eye Child' Robot Adoption Agency as a FactMiners plug-in game.