Flare to MediaWiki to Flare (part 2, make an API call and a Flare TOC)

To get the wiki pages, something has to make the API calls. A program is needed. I decided to build it rather than look for one that fits. But the plan was to use preexisting libraries to access the MediaWiki API if possible. I also decided to go with a Visual Basic project for this next part of the experiment. That decision could have gone many ways. MediaWiki is built on PHP, so that would be a good choice. Java is nice (and I’m familiar with it) and there are plenty of scripting languages which would fit the bill. But, this program is going to run on the same machine as a Flare installation. Flare is a .NET application and so too will be this program to move content to and from Flare projects.

But the real motivation to use Visual Basic was Linq to XML. Once wiki markup was converted to XML, there would probably be more XML manipulation to perform. Linq to XML is good for that. The general idea to get conversion framework started was this:

• Make an API call to get the list of pages on the wiki
• Create a one level Flare TOC based on the list of pages
• Make API calls to get the wiki page markup
• Convert the wiki markup to some kind of XML
• Create Flare topics with the XML

Accomplishing that will not finish the MediaWiki to Flare conversion part of the story. But it will give it a good start and something to work with. Once the basic framework for moving the content is in place, other decisions can be made such as how to handle persisting metadata from the wiki page.

In a previous post, how to create a Flare topic with Linq to XML was demonstrated. So one piece of the puzzle is out of the way. I decided to tackle the easier part of the problem firstly: getting the list of pages and creating the Flare TOC. This was started at the end of the previous post with this API call:

http://localhost/mediawiki-1.19.1/api.php?action=query&generator=allpages

Which returns this:

<?xml version="1.0"?>
<api>
  <query>
    <pages>
      <page pageid="1" ns="0" title="Main Page" />
    </pages>
  </query>
</api>

When I looked at that, TRANSFORMATION screamed in my head. But I wasn’t sold yet. A previous post demonstrated how to perform a transformation to create a Flare topic from XML with XSLT and the approach is roughly the same to create a Flare TOC from XML data. But before that, I figured I should create a Visual Basic project. I created a Console Application project called MediaWikiToFlare. At this point, I planned to run the conversion from a command line or PowerShell.

I know I said I planned to use a preexisting library for the API calls. But just to start, I decided to hard code the REST call. Here is a good example for Yahoo API REST calls: Make Yahoo! Web Service REST Calls With VB.NET. I followed that basic pattern for my initial test. I used Linq to XML to store the returned XML and to make the conversion to a new document.

Imports System
Imports System.IO
Imports System.Net
Imports System.Text
Imports System.Xml
Imports System.Xml.Linq

Module Module1

    Sub Main()
        Dim request As HttpWebRequest
        Dim response As HttpWebResponse = Nothing

        Try  
            request = DirectCast(WebRequest.Create("http://localhost/mediawiki-1.19.1/api.php?format=xml&action=query&generator=allpages"), HttpWebRequest)
            response = DirectCast(request.GetResponse(), HttpWebResponse)

            Dim WikiPagesXML As XDocument = XDocument.Load(response.GetResponseStream())

            Dim FlareTOC As XDocument =


            For Each el In WikiPagesXML.Descendants
                If el.NodeType = XmlNodeType.Element And el.Name = "page" Then
                    FlareTOC.Root.Add( Link=/>)
                End If
            Next

            FlareTOC.Save("C:\WikiToFlare\Project\TOCs\WikiPages.fltoc")

        Finally
            If Not response Is Nothing Then response.Close()
        End Try

    End Sub

End Module

I created the folder structure for the save:

C:\WikiToFlare\Project\TOCs\

And ran the application. A file called WikiPages.fltoc was created in the folder. Here is how the file contents look:

<?xml version="1.0" encoding="utf-8"?>
<CatapultToc Version="1">
<TOCEntry Title="Main Page" Link="Main Page.htm" />
</CatapultToc>