Well, I am throwing in the towel on this. I have made it work, but it just doesn’t work very well. too bad because, as nodes, the stories can be indexed and searched the same way that the rest of the site can. But all I can get from the current setup is a title and a description. I am going to leave the rss items as db items and use Views to search through them.
Tag: aggregator feeds
Search Module – Options and Plugins
Google proved it – Search is key. and the search that comes with Drupal is OK. But not great. I have some search features that I am using by exposing different Views’ filters to the site users. But that really isn’t good either. Melissa Mayer showed the world that simplicity is best (she designed Google’s nearly blank search page)
and that lesson isn’t lost here. I need one way to search everything on my site. and that’s it. that is my quest for today and the last one that I will undertake before I start in on the actual design of the site with CSS and all that fun stuff. From the standpoint of function, this is the last thing to be done.
This is a good page to start with in regards to search. What is cool about the core search feature is the ability to extend the functionality via the checkboxes for additional search mods. —–>
What I really need to be able to search is the Aggregator items that I store. They are DB objects and not nodes which is what makes them tricky.
Drupal to Twitter – Complete (and to Facebook too!)
Wow, I am finally done with this. I now have my categorized RSS feed going straight to a Twitter account and a Facebook account. Both of these accounts are specific to my web site but that doesn’t really matter. And the real key to getting this to work is Twitterfeed.com. This is a free service that will take RSS feeds and consolidate them into Twitter, FB, Linkedin, there are a couple others too, I believe. But Twitter and FB are enough for me.
The Twitterfeed part wasn’t too bad to connect but it did require some tweaking to get it to work with what I have properly. If you decide to use Twitterfeed.com, be aware that the advanced options are more likely to need tweaking than advanced options usually are.
And what do I have?
- 50+ RSS Sources from all over the state of TN in the USA.
- 15 Categories configured in the Drupal Core Aggregator module. I am not using Feeds at all; it isn’t flexible enough for this case.
- 1000 individual news stories that come in every day from these RSS Sources.
- A Drupal View that displays the News items that I categorize as they come in.
- An RSS “Feed” of that View that is available via http://docresource.org/news/24-twitter. This is a part of Views. Click Add, and click Feed.
- A twitter feed – @docresource and FB account – DOCResource that need to get the same info that the View displays
The Twitter module was not needed to do this. Just Views and the Core Aggregator module. There are good reasons to use the Twitter mod; they just don’t apply here.
Aggregators, CRON Jobs and Drupal cleanup.
This was a really involved project. If you use the Aggregator Core module a lot, take a look. I depend on Aggregators more than anything right now and have really had to do some involved work with it. Read on:
I have aggregator needs that the core doesn’t really quite give me. but it does work pretty well. here is what I collect:
- 50+ feeds from various newspapers culled hourly resulting in several hundred articles per day.
- Each RSS source is categorized (automatically, by default in drupal) as z-Uncategorized which corresponds to a CID (in the drupal DB) of 22.
- As the articles come in, I review and categorize them. I have a shortcut to the z-uncategorized category of items. That gives me all the new items, regardless of the source in one place where I can categorize them quickly by clicking on the categorize tab provided by Core. I keep about 10% of the stories that come in.
- Because the newspapers maintain articles in their RSS feeds for a period of time beyond my control, they are readded to drupal’s DB whenever the feed is pulled; but now listed with two categories. There are now two entries for each of these stories with the same IID but a different CID. It looks like this below. There is the default z-uncat… category and the Juvenile category that I chose before the feed was queried again.
- Even thought this looks like one record, it is really two different records in the tables. So, if I look at the aggregator_category_item table, I can see two records for the one IID. One with CID of 22 (the default, z-uncategorized) and the other of whatever I assigned it to. So, I can run a query and delete all with category 22. But, until the newspaper removes it from THEIR feed, it continues to come through.
- I perform a nightly clean up where I delete all the 22s. This occurs when the papers are slow and new items have all been categorized by me.
- Eventually (after a few days for most news sources) the stories are removed from the papers’ RSS feeds and do not get repopulated in Drupal with the default of CID 22. So then I am left with a nice single record in the category that I have assigned it to. By cleaning up every night, I get rid of stale 22s as the newspaper removes them from their RSS feed and I don’t have to think about whether they still have it or not.
This is the cron job that I have to do the clean up.
0 22 * * * /usr/bin/mysql –defaults-file=”/home/xxxx/.my.cnf_cron” -e “DELETE FROM drupal.aggregator_category_item WHERE aggregator_category_item.cid = 22” >>/dev/null 2>&1
The .my.cnf.cron file contains authentication information
[client]
host=localhost
user=crondel
password=*****
The user and password is a mysql specific user I created for this job.
The 0 22 * * * means that it will run at 10 PM EST every night. EST because that is the time zone for the server.
Here are the specific rights for the crondel account name for the drupal DB, named, drupal.
GRANT USAGE ON *.* TO ‘crondel’@’localhost’ IDENTIFIED BY PASSWORD ‘*6E52D2AA6010C379DE1AE3BC559E2416A9A5C513’
GRANT SELECT, DELETE ON `drupal`.`aggregator_category_item` TO ‘crondel’@’localhost’
The account needs SELECT rights to execute the WHERE condition of the SQL statement in addition to the DELETE FROM on the specific table in the DB.
You might ask, why not do all this with Feeds? Well, I did try to do it with Feeds. I spent quite a bit of time with it. But feeds grabs each RSS item as a node. And I could not figure out an easy way to categorize the hundreds of stories per day when they all come in as nodes. And since this DB will eventually be huge with 100k+ stories in a searchable archive, I think that it may be easier to keep it this way. I just had to figure out what to do with the extra 22s. And this solution seems to work.
Ug. This was a pain. And if you want to know more about the subject or I have been unclear, let me know and I’ll try to clarify.
Aggregator categories, Views, Feeds, Panels and a solution
I am very happy today. I finally found a solution to the issue that I was having with Views and Aggregator items from the core mod. Aggregator is core, Views is contrib so this patch doesn’t involve hacking the core but be aware that an updated version of Views might break this again. I’m on Views 7.x.3.5
Here is the issue.
In Views, you can create a view for Aggregator items. But even though Categories is listed as an available field to add/filter against, it is not available unless the category field is used as the default category for that feed. when you go through the items and categorize them manually, that assigned category won’t show up in the View. this patch fixes that.
http://drupal.org/node/498438#comment-7063554
This is what I have. 20 RSS feeds from news sources all over the state. This is a catch all and I am only interested in certain stories. So, I have categories created to group the stories that I want together, regardless of the source. Every item that comes in has a default category of New. That way, I can see all new items together for categorization. This all works. It’s a pain, I have to look at about 40-50 items per day and categorize each one. But it is a big deal because I am dealing with a focused target audience. A niche. And stories that deal with their situation are import and need to be easy to get to.
So, I also need the power of Views. Views gives me much better presentation options when used by Panels than the Blocks interface does. Blocks is clunky and Panels is much better. And Views integrates well with Panels.
Now, you might ask, “why not use the Feeds Module?” well, i did. for hours and hours. and while it will do what I want, it is much more difficult for me (AT MY PRESENT LEVEL OF UNDERSTANDING*) to edit, categorize and tag individual nodes created by Feeds for each new news item. That takes a lot longer. And until I can write a module that does autotagging the way that will work for my type of content, I need to be able to use the Aggregator categories they way they are intended.
So, I implemented the patch. And it does work. I had to do it manually because code has been added since the patch was written to the Views mod and the line numbers are different. But it is really short. I’m on Views 7.x.3.5, Core 7.19. The patch goes into the modules/aggregator.views.inc file within the Views folder.
If you want to see why Views doesn’t work properly with Aggregator ietms read the post. the explanation is in there. This was a really frustrating experience but like all of those types of experiences, I am better off for it.
Views, Feeds, Aggregators and aggravation
I’ve had a few frustrating days with Drupal this week. Hence, the dearth of new posts. A word of warning though: don’t make too many assumptions about Drupal when you’re getting your “sea legs”. There are many, many ways to do things. and some that appear to be promising are frequently blind alleys. I think the best piece of advice that I could give right now would be to KISS – keep it simple stupid. Drupal is already complicated enough. Don’t go out of your way to make things any more difficult than they need to be.
- Learn Views inside and out (it’s the saving grace of Drupal)
- Have a dev/sand environment
- be patient and creative. think around drupal problems and not through them
that’s it for now. going to try to get these feed imports to work the way i need them to work.
Views and Filter Criteria
The views module is one of the greatest strengths in Drupal. But it can be a little intimidating at first. And there isn’t enough documentation available online. but i need to learn it and use it. here is what i needed to do and how i did it.
I have 25 aggregator feeds. I categorize them periodically as they come in from newspapers all around the state. I needed a way to show only the “new” items. So I created a view.
To show only the ones from the current day I needed to create a filter. here is what it looks like
FILTER CRITERIA
the only real problem that i had was how to set the operator. the – 24 hours means go back from the current system time 24 hours as a start point for the view. the operator tells the system to show timestamps of that or greater, which translates to closer to the current time.
Views is a great tool. You have to learn it to use Drupal effectively. Now i am going to learn how to integrate it with panels so I can have the maximum amount of flexibility in displaying my views.
Import/export Aggregator feeds part II
well, i now have it running in prod and there were a few minor differences that i wanted to document.
since i don’t have an X window type desktop, i had to access the phpmyadmin via a remote web browser. so, i had to change the file /etc/conf.d/phpmyadmin.conf in three places to allow connections from all instead of just localhost. i was then able to access the web gui, import the file, and test the aggegator feeds. i had to chanmge som permissions as well to be able to access the file so i could edit it remotely. i could have done it locally via VI Editor, but i hate that thing. it sucks. so after the changes were made i made sure to change the perms back as well as change the file itself so that attempting to access the phpmyadmin gui from the remote host yielded again a “forbidden” error.
Drupal News Aggregator Services
well, i’m trying out different news aggregators. there aren’t too many options but i think that the big problem here is that i am trying to make this easy on myself and not wanting to do the (often) hard work of learning something about drupal. i like drupal and it is as powerful as it gets but it can also be a complete and total pain in the ass when it comes to making it do what you want. that is why i have decided as a long term project to learn css and then php. that’s the best way to really become proficient at drupal.
ok, activity stream seems to be a good way to aggregate content on an individual user basis. not helpful for me. that was easy. i believe that feeds is going to be the way to go. but i have to spend some time with it and learn how to make it work properly.
landie.tech 