Evaluation… Part 3

The final part of the user evaluation (part 3) analyses the responses given on the questionnaire concerning their use of LOs and their response to the metadata generator. Enjoy J

Evaluation Study … part 1

Over the last few days I have been testing the metadata generator with some willing participants (3 in total). There were four parts to this study. The first task set comprised of a series of find and complete tasks aimed at testing the intuitiveness of the interface and terminology. Participants at this stage weren’t privy to help files or detailed information about the applications processes. The following report presents the finding of this part of the study and makes recommendations regarding the interface and terminology used only. The next three parts, auto generation process, uploading to intraLibrary and the questionnaire result, will be forth coming over the next few days. In view of this study I have revisited some of the questionnaires we undertook early on, based on the paper prototyping. We had very few respondents, so statistical analysis was not warranted, however some of the comments made were taken on board at the time. I will be including a brief discussion of these and the relevant aspects and compare, if possible, to the more recent questionnaire results.

 

Releasing the metadata prototype

I’m was intending on releasing the first version of the prototype this week, I suspect it will be end of next week now. Came across a few problems. Well the first isn’t really a problem but it is time consuming and must been done right. The question is what Rights package are we going to release the prototype under? Having been to the Web2Rights events and spoken at some length with Naomi from the project I know how critical this is.

So I decided I best go through the process the Web2Rights team have developed and check everything’s in order. First I checked out the diagnostic tool they have developed. As we have developed a new product the process id fairly straight forward. So I downloaded the appropriate checklist. The main things we have to consider are:

• Any third party Library and tool use – this has already been attributed in the code where applicable. Just need to pull out to the top level the CP notices for the libraries I have used.
• Attribute the authors – I’ve assumed I can refer to the project by name rather than the individual members with LeedsMet being named as the lead institution with funding from JISC.
• JISC’s policies – form what I have read they opt for an open source strategy, please correct me ASAP if I’m wrong about this.

Rather than pen our own licence I would suggest using the GNU which according to the information on Web2Rights is compliant with JISC polices. I also suggest going with their General Public License rather than Lesser General Public License for the reasons stated on their information pages .

Now I just have to cut and paste the header into all the code files and problem solved.

Second problem slightly less legal slightly more complex, I’m talking documentation and support. So far I have the help file developed for the recent repository day. This I think needs a bit more detail. I have also hacked the prototype this afternoon so this same doc is available form the help menu.

While talking Tony through the prototype this morning, with a view to evaluation, it was clear from his comments that its not as obvious as I first thought to know what to do with the application. I had intended to do some popup tips for the fields on the edit metadata panels, some are already there. But it is clear that this is not going to be sufficient. I know this is a beta stage prototype, but I think I need to rethink this before we do some serious evaluation.

So I’ve asked Nick and Tony to be guinea pigs to find out which bits critically need some inline help. This is scheduled over the next couple of weeks.

The final thing is name. What are we to call this application? It is an automatic metadata generator, which is a bit long winded and not so catchy. I keep referring to it as the Autogen which really doesn’t say much about it. Suggests on a post card please.

Repository Tools Taster Day

Today the Streamline project finally got to show off the completed metadata prototype to non-project members. As part of a joint taster session with the Repository and PERsONA projects, we invited members of Leeds Met staff to enjoy a presentation and get hands on with various tools. Over the course of the day various groups were entertained by Nick’s presentation and his amazement at having got twenty odd fans on his Leeds Met Facebook group. Just goes to show the power of social networking.

The majority of people were interested in getting a look inside the repository. I got the impression that at first they thought we were designing the interface to Interlibrary. I mentioned this to John who suggested that we need to devise a clear map of how all these applications and interfaces work and which bits are being developed.

It was a little difficult to get people involved with the metadata tool unless I was there to show them around. The instruction sheet I had put together was a bit long winded given the time people had to look at each of the tools available. Meg, promoter of the eCat application at the start of this project, had a go with it. She seemed suitably impressed, both by the application and how we had incorporated the suggestions she had made when discussing the difficulties with eCat.

Others were impressed with the keyword generation process and interested in the method of text extraction we had used. Generally the response I got from those that tried out the prototype was good. I was unsure, other than Meg, whether the participants all had had previous metadata creation experience, although two were producing learning objects. I think this tool needs to be tested by individuals whose current workflow includes metadata creation rather than those who are new to the experience.

So far no one has added any comments to the blog John has set up for repository tool evaluation. So I can’t comment on that

TEL Day: Questionnaire

The TEL day presented a good opportunity to promote the Streamline project alongside PERSoNA and the Institutional Repository. Nick and I put together a shared questionnaire between these three projects, which I have now done the preliminary analysis on. This report shows the questions I asked and some basic statistics gleaned from them. Nicks findings are available here on the projects respective blogs: PERSoNA and Repository.

We had twenty respondents in total many of whom also indicated that they were willing to participate in the focus groups Meg has mentioned. Unfortunately our first attempt at organising one has been cancelled due to lack of volunteers. Meg suggested that there is a lot going at the moment academically (exam boards and final marking) and that we would be better of in a coupe of weeks.

I’m going to take these results and re-examine them alongside our previous questionnaires. I’m manly look for patterns of positive or negative attitudes and work practices towards the process around learning object creation and re-use. I will post these up in couple of weeks with a more detailed report. I’m also going to have a look at the social networking questions Nick asked in regard to our ideas about the organisation and sharing of learning objects.