<SIGMETRICS@LISTSERV.UTK.EDU>
Mime-Version: 1.0 (Apple Message framework v1081)
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

** Cross-Posted **

Some prima facie critiques from Steve Hitchcock, concerning the MELIBEA =
OA Policy Evaluator:

Begin forwarded message:

> From: Steve Hitchcock <sh94r -- ecs.soton.ac.uk>
> Date: July 15, 2010 5:22:30 AM EDT
> To: boai-forum@ecs.soton.ac.uk
> Cc: SPARC-OAForum@arl.org
> =20
> Reme,    Thank you for bringing this new service to our attention. OA =
policies are vitally important to the development of institutional =
repositories, and services that can highlight and bring attention to =
this development can be valuable.
> =20
> There are a few aspects of the validation aspects of the new MELIBEA =
service that confuse, and possibly trouble, me. The first is the main =
indicator, %OAval, which is the most visible result for a policy. What =
do you expect this will tell people about a given policy? I randomly =
selected a couple of policies, one of which was for my own school, to =
find they each scored about 50%. I would expect these to be among the =
leaders in terms of OA policies, so this seems a surprisingly unhelpful =
score.
> =20
> So what's the explanation? Note that the objects being evaluated are =
institutional OA policies; they are effectively being presented in =
relation to institutional repositories when the policy specifies where =
to archive is an IR with a URL. It seems that the scores include ratings =
for OA publication policy, libre vs gratis OA, publisher pdf, sanctions =
(score if Yes), incentives (score if Yes), etc., some of which an =
institution might specify but which might not apply to an IR =
http://www.accesoabierto.net/politicas/politicas_estructura.php. However =
you weight these factors they are still contributors to the overall =
score, so a policy that is specific to an IR is immediately handicapped, =
or appears to be unless there is more context to understand the scores.
> =20
> Which leads me to another question on the visualisation of the =
validator, and its use of green, gold (and red) in the meter. Do the =
green and gold refer the the classic OA colours? This would be quite =
convenient, since it would appear that the green repository policies I =
mentioned above are achieving almost full scores in the green zone of =
the meter. However, I suspect this cannot be the case, because it would =
assume that institutions must have a green AND gold policy, but not =
simply gold (whatever argument could be put for that).=20
> =20
> It is important that new services should help reveal and promote OA =
policies, as you seek to do, but at the same time not to prejudice the =
development of such policies by mixing and not fairly separating the =
contributing factors, especially where these relate to different types =
of OA.
> =20
> Steve Hitchcock
> IAM Group, Building 32
> School of Electronics and Computer Science
> University of Southampton, SO17 1BJ, UK
> Email: sh94r -- ecs.soton.ac.uk
> Twitter: http://twitter.com/stevehit
> Connotea: http://www.connotea.org/user/stevehit
> Tel: +44 (0)23 8059 7698    Fax: +44 (0)23 8059 2865
> =20
> On 15 Jul 2010, at 08:14, Remedios Melero wrote:
> =20
>> Good mornig!
>> In the last Open Repositories Conference which was held last week in =
Madrid (http://or2010.fecyt.es/publico/Home/index.aspx ) was presented =
in the poster session the project called MELIBEA.
>> MELIBEA (http://www.accesoabierto.net/politicas/)  is a directory and =
  a validator of institutional open-access (OA) policies regarding =
scientific and academic work. As a directory, it describes the existing =
policies. As a validator, it subjects them to qualitative and =
quantitative analysis based on fulfilment of a set of indicators ( =
http://www.accesoabierto.net/politicas/politicas_estructura.php) that =
reflect the bases of an institutional policy.
>> =20
>> Based on the values assigned to a set of indicators, weighted =
according to their importance, the validator indicates a score and a =
percentage of fulfilment for each policy analyzed. The sum of weighted =
values of each indicator is converted to a percentage scale to give what =
we have called the =93validated open-access percentage=94 (see how i t =
is calculated:  =
http://www.accesoabierto.net/politicas/default.php?contenido=3Dacerca ).
>> =20
>> The types of institution analyzed include universities, research =
centres, funding agencies and governmental organizations.
>> =20
>> MELIBEA has three main objectives:
>> =20
>> 	=95 1. To establish indicators that reveal the strong and weak =
points of institutional OA polices.
>> 	=95 2. To propose a methodology to guide institutions when they =
are drawing up an institutional OA policy.
>> 	=95 3. To offer a tool for comparing the contents of policies =
between institutions.
>> The aim is not to be a ranking, but to offer a tool where to aanlyse =
and visualize the weaknesses or strenghts of an institutional OA policy =
based on its wording. It seems something trivial  but accomplishment of =
a policy is based on its terms.
>> Please if you detect any mistake or you would like to make a comment, =
contact me. I will be pleased if you could check your policy, if any, to =
analyse our approach.
>> Best wishes
>> Reme
>> =20
>> =20
>> R. Melero=20
>> IATA, CSIC=20
>> Avda Agust=EDn Escardino 7, 46980 Paterna (Valencia), Spain=20
>> TEl +34 96 390 00 22. Fax 96 363 63 01=20
>> E-mail rmelero -- iata.csic.es=20
>> http://www.accesoabierto.net