The test driver snippet from the last entry hints at the core processing loop that we need for NewsHeap - Iterate over all channels, retrieve their content, parse the result. Rinse, Lather, Repeat.
This post begins to layer some behavior ontop of our current classes, pushing NewsHeap towards useful.
A feed reader should be able to
- Maintain a list of subscribtions
- Routinely query each subscription to see if updated content is present
- Maintain a record of when the content was last checked
- Provide per channel update frequency
The OPML file serves as the list of subscriptions. We can parse it, we can update it, alas we can’t insert or move outlines around yet, but that will come.
To keep track of when channels were last updated, etags if present, update frequency etc, we need to introduce a Subscription. We also need something to manage a list of Subscriptions. A SubscriptionList.
One design issue to wrestle with is the representation of Subscription organization. Foreshadowing our UI requirements to organize our subscription list heirarchically, do we replicate the heirarchy between the subscription list and the OPML? Do we eliminate the OPML altogether and make it available as a import/export format?
Having gone back and forth on this, it strikes me that we can borrow the OPML parser and create a control file format by extending OPML and Outline.
A simple change to the OPML SAX Handler facilitates us setting the class for new outline instances.
def initialize(handlerClass=Outline)
...
@handlerClass = handlerClass
end
def start_element uri, localname, qname, attributes
if (!@opml) # ensure that first element is 'OPML'
...
elsif (localname == 'outline')
o = @handlerClass.new() # Was —> Outline.new()
We make a corresponding change to OPML.readFrom so that we can control the type of Outline.
We can create some subclasses now to hide these details from users of the classes. We need to create Subscripton and SubscriptionList
class Subscription < Outline
def initialize()
@etag = nil
@lastModified = nil
@xmlURL = nil
@lastGet = nil
super()
end
def etag=(anEtag)
@attributes['etag'] = anEtag
end
def etag()
@attributes.fetch('etag',nil)
end
You will note that we harden the interface for the Subscription class, adding accessors for elements such as etag.
SubscriptionList is even easier
class SubscriptionList < OPML
...
def readFrom(aSource)
super(aSource,Subscription)
end
This will ensure that when we do
subList = SubscriptionList.new
subList.readFrom(File.new("somecontrolfile.xml"))
that all instances in the subscription list will be of class Subscription
A few unit tests and the addition of an iterator in OPML, and we’re done.
Driver
Let’s bring this together with a driver program.
First, some top matter and initialization
opmlFile = ARGV[0] || "myChannels.opml"
# read the control file
subscriptionList = NewsHeap::SubscriptionList.new()
subscriptionList.readFrom (File.new("./control/control.xml",File::CREAT|File::RDONLY))
# read the opml
opml = OPML.new
opml.readFrom(File.new(opmlFile))
parser = RssParser::new()
fetcher = HttpGetter::new()
The preceding checks for a control file in ./control/control.xml and creates it if not found.
Start looping over every entry in the OPML file, checking the SubscriptionList for an entry.
# for every outline
begin
opml.each_outline { | outline |
url = outline.attributes["xmlUrl"]
# check control file
subscription = getSubscription(subscriptionList,url)
result = {}
etag = subscription.etag
lastModified = subscription.lastModified
data = fetcher.readData(url,result,etag,lastModified,NewsHeap::Control::AGENT)
if (newData?(result,subscription))
result = parser.parse(data,result)
end
printIt(result)
# update the control entry
subscription.etag = result['etag']
subscription.lastModified = result['modified']
subscription.lastGet = Time.now()
}
rescue => bang
print_backtrace(bang)
end
# update control data
subscriptionList.persist("./control/control.xml")
We grab the etag and lastModified attributes from the subscription and pass them to our HTTPFetcher. The fetcher may return a 304 Not Modified if we’ve already got current content. The fetcher always updates the etag and last modified, so we can check if a new entry is present. If so, we’ll pass it to the parser. Update the control entries and when we’re all done, we persist the control file for use in the next go around.
One final enhancement makes this actually usable
...
etag = lastModified = nil if !cached?(url) # <=== Check the cache for this URL - don't modify headers if no cache hit
data = fetcher.readData(url,result,etag,lastModified,NewsHeap::Control::AGENT)
# update and cache results
cache(data,url) if data # <=== cache the data so if we get a 304, we still have content to display
if (newData?(result,subscription))
result = parser.parse(data,result)
end
Summary
We have a control file to store details of our subscriptions and support for bandwidth friendly behavior.
What we’re missing is per-channel update frequency and some behavior to rearrange the control file to reflect heirarchies and groups.
In the next round, we’ll examine some GUI options, make a decision and implement the basic 3-pane viewer.
The existing code still needs to be cleaned up - camelCase vs canonical_ruby, organization etc can all be better. I’m adding unit tests as we go, so we can layer in some good feelings.
Download file
