ConferenceCall 2013 01 31

= OntologySummit2013: Virtual Panel Session-03 - Thu 2013-01-31 =

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"

Summit Track Title: Track-A: Intrinsic Aspects of Ontology Evaluation

Session Topic: Intrinsic Aspects of Ontology Evaluation: Practice and Theory


 * Session Co-chairs: Dr. SteveRay (CMU) and Dr. LeoObrst (MITRE) ... [ intro slides ]

Panelists / Briefings:


 * Ms. MariaPovedaVillalon (Universidad Politécnica de Madrid), Dr. MariCarmenSuarezFigueroa (Universidad Politécnica de Madrid) and Dr. AsuncionGomezPerez (Universidad Politécnica de Madrid) - "A Pitfall Catalogue and OOPS!: An Approach to Ontology Validation" ... [ slides ]
 * Dr. SamirTartir (Philadelphia University, Amman, Jordan), Dr. IsmailcemBudakArpinar (University of Georgia), Dr. AmitSheth (Wright State University) - "Ontology Evaluation and Ranking using [[OntoQA]]" ... [ slides ]
 * Dr. JesualdoTomasFernandezBreis (Universidad de Murcia), Ms. AstridDuqueRamos (Universidad de Murcia), Dr. RobertStevens (University of Manchester), Dr. NathalieAussenacGilles (Institute de Recherche en Informatique de Toulouse (IRIT), Université Paul Sabatier) - "The OQuaRE Framework for Ontology Evaluation" ... [ slides ]

Archives:


 * Abstract
 * Agenda
 * Prepared presentation material (slides) can be accessed by clicking on each of the title links below:
 * [ 0-Chair ] . [ 1-Poveda ] . [ 2-Tartir ] . [ 3-DuqueRamos ]
 * transcript of the online chat during the session
 * Audio recording of the session ... [ 1:53:10 ; mp3 ; 12.95 MB ]
 * its best that you listen to the session while having the respective presentations (linked above) opened in front of you. You'll be prompted to advance slides by the speaker.
 * Additional Resources

Abstract
OntologySummit2013 Session-03: "Intrinsic Aspects of Ontology Evaluation: Practice and Theory" - intro slides

This is our 8th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.

During this OntologySummit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communiqué.

At the Launch Event on 17 Jan 2013, the organizing team provided an overview of the program, and how we will be framing the discourse around the theme of of this OntologySummit. Today's session is one of the events planned.

In this 3rd virtual panel session of the Summit, we focus on theory and practice for intrinsic aspects of ontology evaluation. Our speakers will present a number of approaches and frameworks for evaluating the quality of ontologies, and some theoretical discussion of what constitutes intrinsic evaluation. Our main goal in this virtual session is to begin to lay out the criteria for intrinsic evaluation of ontologies, some possible metrics, and the rationale for these. We hope that all of the participants in the open discussion and chat will join us in helping to flesh out intrinsic evaluation criteria and their dimensions.

More details about this OntologySummit is available at: OntologySummit2013 (homepage for this summit)

Briefings:

 * Ms. MariaPovedaVillalon (Universidad Politécnica de Madrid), Dr. MariCarmenSuarezFigueroa (Universidad Politécnica de Madrid) and Dr. AsuncionGomezPerez (Universidad Politécnica de Madrid) - "A Pitfall Catalogue and OOPS!: An Approach to Ontology Validation" ... [ slides ]
 * Abstract: ... One of the advantages of using methodologies for building ontologies is to improve the quality of the resultant ontology. However, due to the difficulties involved in ontology modelling, such quality is not totally guaranteed. These difficulties are related to the inclusion of anomalies or worst practices. Our approach contributes to the ontology validation activity by (1) providing a catalogue of common worst practices, which we call pitfalls, and (2) proposing a web-based tool, called "OOPS!". This approach will help developers in two following ways: (1) to avoid the appearance of pitfalls in ontologies, and (2) to improve ontology quality by automatically detecting potential errors.


 * Dr. SamirTartir (Philadelphia University, Amman, Jordan), Dr. IsmailcemBudakArpinar (University of Georgia), Dr. AmitSheth (Wright State University) - "Ontology Evaluation and Ranking using [[OntoQA]]" ... [ slides ]
 * Abstract: ... Ontologies form the cornerstone of the Semantic Web and are intended to help researchers to analyze and share knowledge, and as more ontologies are being introduced, it is difficult for users to find good ontologies related to their work. Therefore, tools for evaluating and ranking the ontologies are needed. In our talk, we present [[OntoQA]], a framework that evaluates ontologies related to a certain set of terms and then ranks them according a set of metrics that captures different aspects of ontologies. Since there are no global criteria defining how a good ontology should be, [[OntoQA]] allows users to tune the ranking towards certain features of ontologies to suit the need of their applications.


 * Dr. JesualdoTomasFernandezBreis (Universidad de Murcia), Dr. AstridDuqueRamos (Universidad de Murcia), Dr. RobertStevens (University of Manchester), Dr. NathalieAussenacGilles (Institute de Recherche en Informatique de Toulouse (IRIT), Université Paul Sabatier) - "The OQuaRE Framework for Ontology Evaluation" ... [ slides ]
 * Abstract: Many Software Engineering methods have been adapted and applied to ontology engineering in the last decades. This has not been the case so far of ontology evaluation despite the availability of international software quality standards. In contrast, the increasing importance of ontologies has resulted in the development of a large number of ontologies, but there is a lack of mechanisms for guiding users in making informed decisions on which ontology to use under given circumstances make hard to ontology users and tool developers selecting ontologies to be used and reused.  We propose a framework named OQuaRE for evaluating the quality of ontologies based on the SQuaRE standard for software quality evaluation. The main objective of OQuaRE is to provide an objective, standardized framework for ontology quality evaluation in order to identify the strengths and weaknesses of ontologies. OQuaRE aims at helping users in making informed decisions and ontology experts in evaluating the quality of their ontologies with the support of automatically calculated metrics. We will not only describe the current version of the framework but also the results and evaluation of its application by external experts.

Agenda
OntologySummit2013 - Panel Session-03


 * Session Format: this is a virtual session conducted over an augmented conference call


 * 1. Opening (co-chair) - SteveRay / LeoObrst [10 min.] ... [ slides ]
 * 2. Panel briefings - MariaPovedaVillalon, SamirTartir, AstridDuqueRamos - [25 min. each]
 * 3. Q & A and open discussion [All: ~30 min.] -- please refer to process above
 * 4. Wrap-up / Announcements - (co-chairs) - LeoObrst / SteveRay [5 min.]

Proceedings:
Please refer to the above

IM Chat Transcript captured during the session:

see raw transcript here.

(for better clarity, the version below is a re-organized and lightly edited chat-transcript.) Participants are welcome to make light edits to their own contributions as they see fit.

-- begin in-session chat-transcript --

[08:23] PeterYim: Welcome to the

= OntologySummit2013: Virtual Panel Session-03 - Thu 2013-01-31 =

Summit Theme: Ontology Evaluation Across the Ontology Lifecycle


 * Summit Track Title: Track-A: Intrinsic Aspects of Ontology Evaluation

Session Topic: Intrinsic Aspects of Ontology Evaluation: Practice and Theory


 * Session Co-chairs: Dr. SteveRay (CMU) and Dr. LeoObrst (MITRE)

Panelists / Briefings:

- Ms. MariaPovedaVillalon (Universidad Politecnica de Madrid) - Dr. MariCarmenSuarezFigueroa (Universidad Politecnica de Madrid) - Dr. AsuncionGomezPerez (Universidad Politecnica de Madrid)
 * "A Pitfall Catalogue and OOPS!: An Approach to Ontology Validation"

- Dr. SamirTartir (Philadelphia University, Amman, Jordan) - Dr. IsmailcemBudakArpinar (University of Georgia) - Dr. AmitSheth (Wright State University)
 * "Ontology Evaluation and Ranking using [[OntoQA]]"

- Dr. JesualdoTomasFernandezBreis (Universidad de Murcia) - Ms. AstridDuqueRamos (Universidad de Murcia) - Dr. RobertStevens (University of Manchester) - Dr. NathalieAussenacGilles (Institute de Recherche en Informatique de Toulouse (IRIT), Universite Paul Sabatier)
 * "The OQuaRE Framework for Ontology Evaluation"

Logistics:


 * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_01_31


 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)


 * Mute control: *7 to un-mute ... *6 to mute

if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.
 * Can't find Skype Dial pad?
 * for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad"
 * for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,)

Proceedings:
[09:06] anonymous1 morphed into MariaPovedaVillalon

[09:14] anonymous1 morphed into MeganKatsumi

[09:16] anonymous1 morphed into CarmenChui

[09:19] anonymous morphed into JimDisbrow

[09:20] anonymous morphed into JesualdoTomasFernandezBreis

[09:20] JesualdoTomasFernandezBreis: Hello all

[09:24] anonymous morphed into MohammadAqtash

[09:25] anonymous1 morphed into DavidMakovoz

[09:26] MeganKatsumi1 morphed into MichaelGruninger

[09:28] LeoObrst: Hi, Jesualdo, Maria, Mohammad and all!

[09:28] anonymous morphed into SamirTartir

[09:28] MariaPovedaVillalon: Hi!

[09:28] LeoObrst: Hi, Samir!

[09:28] MariCarmenSuarezFigueroa: Hello all....

[09:30] MohammadAqtash: Hi LeoObrst!

[09:30] astridduque morphed into AstridDuqueRamos

[09:31] anonymous morphed into MikeDenny

[09:31] anonymous morphed into DougFoxvog

[09:31] anonymous1 morphed into TorstenHahmann

[09:32] anonymous1 morphed into ClarePaul

[09:32] anonymous morphed into JosephTennis

[09:32] anonymous morphed into IsmailcemBudakArpinar

[09:34] anonymous morphed into JamesOdell

[09:35] RamSriram: I do have a problem viewing slides on a Mac (using VNC). It seems to work on a PC.

[09:50] SteveRay: @Ram: My theory remains that this problem relates to the latest version of Java, which somehow keeps the VNC from working properly.

[09:52] ToddSchneider: Steve, Ram, I checked and the browser has the Java plug-in disabled.

[09:36] anonymous morphed into JoaoPauloAlmeida

[09:38] PeterYim: == SteveRay opens the session ... see: [0-Chair] slides

[09:39] List of members: AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, BobbinTeegarden, BobSchloss, IsmailcemBudakArpinar, CarmenChui, ClarePaul, DavidMakovoz, DavidLeal, DougFoxvog, FabianNeuhaus, FranLightsom, HensonGraves, JamesOdell, JesualdoTomasFernandezBreis, JimDisbrow, JoaoPauloAlmeida, JoelBender, JosephTennis, KenBaclawski, LeoObrst, MariaPovedaVillalon, MariCarmenSuarezFigueroa, MarkFox, MatthewWest, MeganKatsumi, MichaelGruninger, MikeDenny, MikeDean, MohammadAqtash, PeterYim, RamSriram, SamirTartir, SteveRay, TerryLongstreth, ToddSchneider, TorstenHahmann, vnc2

[09:42] DuaneNickull: Good day

[09:42] PeterYim: Hi Duane, welcome to the session

[09:42] DuaneNickull: Please to be here.

[09:45] PeterYim: == MariaPovedaVillalon presenting ... see: [1-Poveda] slides

[09:45] anonymous1 morphed into TrishWhetzel

[09:52] SteveRay: We are now on slide 7

[09:54] BobSchloss: As I listen to the approach Maria has started doing, it reminds me of some work I did with my colleague Achille Fokoue-Nkoutche in the very early days of the XML Schema language. We released a tool called IBM XML Schema Quality Checker through the IBM alphaWorks program, it was very widely used because this kind of document / message / vocabulary modeling was unfamiliar to a lot of people, and top quality tools for construction of these XML Schemas (such as from companies such as Altova, Progress Software, IBM Rational) were still not widely available.

[09:56] BobSchloss: Equally importantly, guidelines, best practices and patterns for XML Schema development, which were later compiled by a number of people, were not yet documented... so our tool warned people when they were using a construct that was strictly legal but might limit evolvability of their schema or reuse by others.

[09:56] BobSchloss: [I have to leave for another meeting... Will review all slides and the recording of this chat later. Thanks all]

[09:57] MariCarmenSuarezFigueroa: @BobSchloss, interesting work.

[09:56] SteveRay: @Maria: Question for later: To recognize your Pitfall #5, it would seem that you would need to know the intent of a term such as isSoldIn and isBoughtIn. How does your automated tool do this?

[09:57] MariCarmenSuarezFigueroa: @SteveRay: at this moment our tool OOPS! detect in an automated way a subset of the pitfalls in the catalogue

[09:58] MariCarmenSuarezFigueroa: @SteveRay for those detected by OOPS! there are different approaches as Maria is explaining

[09:59] MariCarmenSuarezFigueroa: (is going to explain in next slides)

[09:59] DougFoxvog: @Steve. The inverse relationship between isSoldIn & isBoughtIn can be determined to be inconsistent with having the argument types reversed. By noting that the argument types match (arg1<=>arg1 & arg2<=>arg2) one can suggest that the error is in calling it an inverse relationship instead of being in mis-assignment of argument types.

[10:01] SteveRay: @MariCarmenSuarezFigueroa, DougFoxvog: Ah yes, I see now - Domain and Range mismatch.

[10:04] DougFoxvog: Slide 14 suggests possible symmetric or transitive properties if there are equal domain & range. Such suggestions should not be made if the relations are already defined as asymmetric or functional.

[10:04] MariCarmenSuarezFigueroa: yes @DougFoxvog

[10:02] MariCarmenSuarezFigueroa: OOPS! is available at http://www.oeg-upm.net/oops

[10:00] anonymous1 morphed into BruceBray

[09:59] SamirTartir: Hello Dr. @Arpinar, @MohammadAqtash... Nice to see you here.

[10:06] SamirTartir: Some definitions I will be using in the my presentation can be found in this paper: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4338348&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D4338348

[10:06] TerryLongstreth: Maria- have you considered running OOPS! against public domain ontologies, and publishing the resulting evaluations?

[10:07] MariCarmenSuarezFigueroa: @TerryLongstreth: thanks for the suggestion. Yes, we have already made an experiment in this sense; and our idea is to evaluate more available ontologies in order to see the current state of public ontologies.

[10:09] JesualdoTomasFernandezBreis: @MariCarmenSuarezFigueroa: evaluating OBO and BioPortal ontologies might be interesting

[10:10] MariCarmenSuarezFigueroa: @Jesualdo, thanks! We have already evaluate a subset of OBO and BioPortal ontologies, but our plan is to made the evaluation for more

[10:10] ToddSchneider: Can the OOPS! source code be obtained?

[10:11] MariCarmenSuarezFigueroa: @ToddSchneider: at this moment the source code is not available.

[10:12] ToddSchneider: Maria, Too bad. Can your team consider working with OOR initiative?

[10:13] MariCarmenSuarezFigueroa: @ToddSchenider: we are analysing how to proceed with our code. Maybe we can consider your suggestion

[10:13] MariaPovedaVillalon: we will provide web services soon Todd

[10:14] ToddSchneider: Maria's, For performance reasons having a more 'local' instance of OOPS! would be optimal.

[10:15] MariCarmenSuarezFigueroa: @Todd, yes you are right. As mentioned, we are now analysing different options for our code :-D

[10:15] MariaPovedaVillalon: @ToddSchneider you are right, as Mari Carmen said we need to think about to to proceed with the code

[10:15] MariaPovedaVillalon: @Todd I totally agree with the idea of sharing the code

[10:16] MariaPovedaVillalon: @Todd as first step we are creating the web services so that everybody can include the features in their code, but sure, we also need to share the code

[10:12] LeoObrst: @Maria: 3 questions: 1) These are all OWL/RDF ontologies; are you considering other languages, e.g., Common Logic? 2) What if the ontology you are evaluating contains imports or references to other ontologies, do you track down these and evaluate them? 3) What is "P9. Missing basic information"?

[10:13] AmandaVizedom: @Maria: Very nice. I have seen several large projects spend extensive effort in developing their own, in-house versions of something like this approach. This speaks to the relevance of the approach to real work-flows. However, due to lack of resources and/or expertise, those in-house versions usually end up not as good as OOPS! appears to be. They also often generate repeating cycles of management or collaborator doubt, as their developers cannot point to independent grounding and acceptance. OOPS! seems to me like a valuable contribution to operational use of ontologies.

[10:14] MariCarmenSuarezFigueroa: @Amanda, thank you for your comment.

[10:12] TrishWhetzel: Of the ontologies in BioPortal, do you find errors correlated to any of the groups, such as OBO Foundry or UMLS, and/or ontology format .. which is also somewhat an indicator of ontology design patterns?

[10:15] JesualdoTomasFernandezBreis: @TrishWhetzel: in another project we have done a systematic analysis of the labels of bioportal ontologies and we found some problems with formats and availability of some files

[10:19] TrishWhetzel: @Jesualdo Thanks, I'm interested in these issues

[10:25] JesualdoTomasFernandezBreis: @Trish: I will send you an email with the details

[10:16] MeganKatsumi: @Maria: Sorry if you mentioned this already, but how do you decide if a particular characteristic qualifies as a pitfall?

[10:17] MariaPovedaVillalon: @Megan we have observed the pitfalls we list as errors in ontologies when we manually analyzed them, however, the same "characteristic" might not be an error in other ontology, so at the end the user decide. Sometimes the error does not need to be checked but that is not always the case.

[10:15] AmandaVizedom: @Maria: A few questions: (1) Is it correct that OOPS! works specifically on OWL? Is it further narrowed to specific dialects (such as DL)? Does your group have any plans or interests in extending to other languages (for example, Common Logic?)

[10:19] DougFoxvog: @Amanda, re your first question. OOPS! only accepted OWL RDF/XML when I looked at it in December.

[10:16] AmandaVizedom: @Maria: (2) Can OOPS! detect errors (or warnings/suggestions) based on general logical entailments? Does OOPS! make use of, or contain, a general OWL reasoner?

[10:19] MariaPovedaVillalon: @Amanda we think about leaving that decision to the user

[10:20] MariaPovedaVillalon: @Amanda reasoners do already exist and we don't want to rethink the wheel, however we can benefit from them to detec more pitfall but the computational price will be too high

[10:34] MariaPovedaVillalon: @Amanda in summary, at any case our idea is that we should leave the decision of using reasoners to the user (maybe a checkbox in OOPS!) or point to existing reasoners giving the user some guidelines about which things to check and common errors

[10:22] AmandaVizedom: @Maria (3) Have you run into any difficulties concerning what seem to be pitfalls and the behavior of OWL reasoners? An example that comes to mind is the fact if a property is applied used to relate two things, one of which is not stated to satisfy the range requirements, for example, it will be inferred that the thing *does* meet the range requirements. But in some (many?) cases, the omission is actually indicative of an error. Would OOPS! treat as a pitfall or warning the fact that the thing in the range is not stated to meet the range requirements?

[10:25] MariaPovedaVillalon: @AmandaVizedom excuse me, if I'm not wrong you talk about instances, right now OOPS! only looks at the schema level

[10:26] MariaPovedaVillalon: @Amanda have I answered your question? I do not think I understood it properly, maybe...

[10:32] AmandaVizedom: @Maria: That's fine. I think that last question may be confusing if we are accustomed to working at different levels of language expressiveness. If you are working in DL it may be a instance-level issue; less so if working expressiveness beyond DL. Thanks!

[10:17] PeterYim: @MariaPovedaVillalon @MariCarmenSuarezFigueroa - we are contemplating on doing a hackathon exercise, it will be great if you team can join us in that effort (we have yet to refine on what exactly would that "hackathon" entail though, so participant input are solicited)

[10:17] MariCarmenSuarezFigueroa: @Peter, yes, count with us

[10:18] PeterYim: @MariCarmenSuarezFigueroa - fantastic! thank you.

[10:18] MariaPovedaVillalon: @Peter sure :)

[10:19] PeterYim: Thanks, Maria.

[10:19] KenBaclawski: @Maria: I built a system very similar to yours back in 2004. I called the problems with an ontology symptoms rather than pitfalls, but it is the same idea. It used a rule-based approach which was extendible. One interesting feature was that the symptoms that were generated were in a Symptom ontology and we performed reasoning on the symptoms that were generated to find relationships among symptoms since we found that a single error can generate many symptoms (which, by the way, is the reason for using the word "symptom"). Here is a reference to the paper: K. Baclawski, C. Matheus, M. Kokar and J. Letkowski. Toward a Symptom Ontology for Semantic Web Applications. In ISWC'04, pages 650-667. Lecture Notes in Computer Science 3298:650-667. Springer-Verlag. (2004)

[10:20] MariCarmenSuarezFigueroa: @KenBaclawski, thanks for the reference.

[10:21] MariaPovedaVillalon: @KenBaclawski, thank you very much for the reference, sounds really interesting and familiar what you said about one error many symptoms...

[10:13] DougFoxvog: (re. Maria's slide#14) One can suggest transitivity if the domain is a subclass of the range. They need not be equal.

[10:28] MariaPovedaVillalon: @DougFoxvog we check what you said about suggestion in slide 14

[10:29] MariaPovedaVillalon: @DougFoxvog we fixed the errors it had in December, OOPS! was supposed to check it and if everything is fine it should work by now

[10:30] JesualdoTomasFernandezBreis: I think tools like OOPS! are fundamental for ontology engineers, thanks for your work!

[10:31] MariaPovedaVillalon: Thank you @Jesualdo

[10:31] MariCarmenSuarezFigueroa: @Jesualdo, thank you very much for your comment!

[10:12] PeterYim: == SamirTartir presenting ... see: [2-Tartir] slides

[10:14] anonymous1 morphed into DennisWisnosky

[10:14] anonymous1 morphed into YuvaTarunVarmaDatla

[10:17] anonymous2 morphed into NathalieAussenacGilles

[10:20] JosephTennis: can someone extract all the citations being shared and put them in one spot?

[10:21] SteveRay: @Joseph: There is one spot we are collecting references. Amanda can say more about that (and did in an earlier session).

[10:22] JosephTennis: sweet! thanks!

[10:23] SteveRay: @Joseph: I should add that it is our shared responsibility to populate it.

[10:28] ToddSchneider: Samir, Would you provide the definitions for each of the variables in your metrics in the chat?

[10:28] AmandaVizedom: @Steve and @Joseph I will post a note to the summit list under the {biblio} subject soon. I've been away for a bit and have begun getting the Zotero library caught up. Meanwhile, the library itself is at https://www.zotero.org/groups/ontologysummit2013/items

[10:33] DougFoxvog: @Samir: many of your metrics are metrics for knowledge bases. It might be useful to distinguish the two classes of metrics, and have different scores for ontologies (which define types and relations) and knowledge bases (which define individuals and provide information about the individuals by asserting relations that apply to them).

[10:36] DougFoxvog: @Samir: I see that you do distinguish multiple ranking types in slides 16 & 17. But defining different sets of rankings for different types of KBs or ontologies might be useful.

[10:34] LeoObrst: @all: I think these evaluation tools and metrics would be very useful in the Open Ontology Repository (OOR). Perhaps the speakers would like to join our OOR group and provide potential services?

[10:36] MariCarmenSuarezFigueroa: @Leo, yes, we can consider to join the OOR group and see how we can contribute to it

[10:40] MariaPovedaVillalon: @Leo where can we find information to join and contribute to OOR?

[10:42] ToddSchneider: Maria, http://ontolog.cim3.net/cgi-bin/wiki.pl?OpenOntologyRepository

[10:43] MariaPovedaVillalon: thanks

[10:39] LeoObrst: @Samir: can you provide definitions for your variables?

[10:43] SamirTartir: @ToddSchneider & @LeoObrst: There is a large number of variables used. E.g. A set of classes, C., A set of relationships, P. An inheritance function, Hc. A set of class attributes, Att.

[10:44] SamirTartir: @ToddSchneider & @LeoObrst: The definitions are all included in the paper I referenced right before I started presenting.

[10:45] SamirTartir: I will be more than happy to send you the paper if you'd like.

[10:45] ToddSchneider: Samir, I missed that reference.

[10:46] SamirTartir: Todd, here it is again: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4338348&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D4338348

[10:46] ToddSchneider: Samir, Great. Thank you.

[10:47] TorstenHahmann: @Samir: before weighting, are the various metrics standardized (to values between 0 and 1, for example)? Otherwise two metric with equal weight may still influence the total score differently

[10:47] SamirTartir: @Torsten: Yes.

[10:50] SamirTartir: @DougFoxvog: Not sure what you mean here. Maybe discuss this after the current speaker finishes.

[10:51] TorstenHahmann: @Samir: I take your answer as they are standardized.

[10:53] SamirTartir: @Torsten. Sorry for not being clear. Yes they are standardized.

[10:53] TorstenHahmann: @Samir: thanks.

[10:38] PeterYim: == AstridDuqueRamos presenting ... see: [3-DuqueRamos] slides

[10:53] JosephTennis: This was great! Wish I could stay longer. I look forward to using some of these metrics for my work. One question I have is what do these metrics look like on different versions of the same ontology? Do they change dramatically? Or do they not change much at all? It might be something we can look at here. In case you're interested in my work on versioning, you can check out my page: http://joseph-t-tennis.squarespace.com/research-streams/ Perhaps we can collaborate?

[10:53] JosephTennis: ciao!

[10:52] AmandaVizedom: @Astrid (actually, this applies to @Samir's presentation as well): your approach measures some numeric/ topographic qualities of an ontology, such as depth of inheritance, breadth of relationships, number of ancestors. I have seen many cases in which ontology teams, or projects of which they are a part, are required to report such metrics upward to management, and are held in some sense accountable for them, but it is very hard to see whether they are actually indicative of quality (and if so how). It may be that they are meaningful given some interpretation, given some requirements in place, or under some other conditions. What do they mean, in your view?

[10:54] TorstenHahmann: +1 to @Amanda's comment: that should be one of the goals of this track (in my opinion)

[10:54] JesualdoTomasFernandezBreis: @Amanda: one of the next slides has some comments I think related to yours

[10:54] SteveRay: @Amanda & @Astrid: Amanda, I think you are precisely raising the question about intrinsic vs. extrinsic evaluation. Management often cares about the latter more than the former, and sometimes at the expense of attention to the former.

[10:57] SamirTartir: @Steve, @Amanda & @Astrid: It's relevant to each user or scenario. I think @Steve 's comment is right on target.

[10:58] LeoObrst: @Steve, Amanda, and all: Yes, because the current application is local and of highest priority to management: extrinsic typically is more valued.

[10:59] AmandaVizedom: @SteveRay: Although I'm very interested in the intrinsic/extrinsic question, I see *this* question a bit differently. Staying within the intrinsic evaluation topic, there is an independent question about which, of the many intrinsic characteristics an ontology may be said to have, are actually measurements of quality. That a quality exists and can be measured, or even that it has some intuitive or aesthetic appeal, is not enough to establish that it is an aspect of ontology *quality*. The question is: Are these? And if so, why?

[11:02] DougFoxvog: +1 @Amanda. many of the mentioned characteristics are *features* of the ontologies. Whether they are measures of *quality* may in some instances be context dependent.

[11:03] TorstenHahmann: I agree with @Doug: independent of whether intrinsic metrics are valued by management, we have to figure whether they are correlated to the intended qualities

[11:03] LeoObrst: @Amanda: perhaps more to your point, an approach such as OntoClean that uses ontological analysis more clearly may have higher real value/quality, but is not necessarily immediately understandable to management, though application value can be demonstrated.

[11:00] SteveRay: We could discuss (at a later time) whether ontology development environments could hard-wire evaluation during the ontology development process.

[11:00] SteveRay: @Amanda: Need to think about your question in a few minutes.

[11:03] AmandaVizedom: @Steve, et al., it may also be that *extrinsically* relevant characteristics are so consistently relevant in some domain / context of application that folks trained in that context believe them to be *intrinsic*. I do not mean to pre-judge the question for the characteristics I mentioned. Rather that I would be interested in the presenters' thoughts on those, and whether they can offer particular reasons for considering those intrinsic measurements to be measures of *quality*.

[11:01] MatthewWest: An even better approach is to have a develop method that avoids the quality problems.

[11:01] PeterYim: == Q&A and Open Discussion ...

[11:01] PeterYim: question for all panelists - how do some of the more rigorously developed ontologies (like BFO, DOLCE, PSL, SUMO, CYC, etc.) fare, when put through your evaluation system/tool; anyone tried? observations & insights gained?

[11:15] MariCarmenSuarezFigueroa: We have already evaluate DOLCE with OOPS! (among other established ontologies)

[11:17] TerryLongstreth: @MariCarmenSuarezFigueroa - is there a URL for the results of the DOLCE evaluation?

[11:19] MariCarmenSuarezFigueroa: @terry, results are not available yet

[11:15] JesualdoTomasFernandezBreis: We are in the process of evaluating all the BioPortal ontologies with OQuaRE, but we do not have the results yet

[11:21] AmandaVizedom: Following up on Peter's question: For all panelists: What are the expressivity constraints or expectations of these tools? Are they limited to DL ontologies? OWL-Full? Has anyone applied their techniques to ontologies represented in FOL or higher languages?

[11:22] MariCarmenSuarezFigueroa: (re. Peter's follow-up question on whether they had tried how the tools scale with larger ontologies like SUMO or CYC) We did not try with Cyc yet, for example.

[11:03] anonymous1 morphed into HashemShmaisani

[11:03] DuaneNickull: (ref. the reverb/echo when DougFoxvog tried to patch in) Nice audio effects

[11:03] MariaPovedaVillalon: :)

[11:03] DuaneNickull: Very Dr. Who - ish

[11:04] DuaneNickull: Exterminate, exterminate, exterminate.....

[11:04] MariCarmenSuarezFigueroa: :O

[11:04] AmandaVizedom: Audio sounds like we have fallen down the rabbit hole!

[11:04] BobbinTeegarden: @DougFoxvog: 'context dependent'... or, in the eye of the beholder?

[11:05] MatthewWest: Some very interesting presentations, but I'm afraid I have to go now.

[11:05] anonymous morphed into AsuncionGomezPerez

[11:08] JimDisbrow: Steve's first point of a measurement being "well-designed" is: "Proper use of various relations found within an ontology". This has been an issue that has been sorely underrepresented, but may now be breaking through - as demonstrated by the presentations. Insertion of reflexivity in relationships, however, was not mentioned as a criteria. Is there any progress in this ontological concepts implementations? (ref. below [11:28])

[11:11] LeoObrst: @Doug: I think Samir's analysis of both ontology and KB are very useful for ontologists, even though KBs will potentially be different across applications, companies, etc.

[11:12] DougFoxvog: @Leo: I agree. But I'm suggesting that these distinctions should be identified.

[11:15] TerryLongstreth: @Leo - I'm not convinced that there's an objective procedure for separating the two. Linnean classification requires a 'type instance' to fully describe a species type. Would that be in the Ontology, or in the KB?

[11:15] SamirTartir: Thanks Leo. Doug: I agree that it might be useful.

[11:18] SamirTartir: (re. Amanda's positive verbal remark about the relationship diversity metric) Thanks @Amanda

[11:19] LeoObrst: @Terry: to your point, that is an issue. E.g., usually classes are considered universals, and instances particulars, but some ontologies (and metaphysics) don't makes those distinctions, e.g., identifying all "ontology" notions as particulars (e.g., tropes, etc.)

[11:20] DougFoxvog: @Terry: OWL DL does not allow meta-classes, so that the instances of species, genus, bio-kingdom, etc. are classes, themselves. A system that merely defines these, their hierarchy, and relations that may apply to them would be, imho, an ontology. However, if data is provided about these taxons (geological range, endangerment, diet, etc.), then it would be a KB, even though what is being described are themselves classes.

[11:22] TerryLongstreth: Then I think evaluation has to include the KB if it's required for full interpretation of the ontology

[11:22] LeoObrst: @Terry: I agree. Both need to be evaluated.

[11:28] JimDisbrow: @Steve: In your first slide, your first point of a measurement for being well-designed is: "Proper use of various relations found within an ontology". This has been an issue that has been sorely underrepresented, but may now be breaking through - as demonstrated by the presentations. Insertion of reflexivity in relationships, however, was not mentioned as a criteria. Similarly, there was no mention of an active "not" verb (not just the English negation term), concatenated into the middle term of the OWL "triple". A question for the presenters: Is there any progress in implementing these ontological concepts?

[11:31] SteveRay: Anybody want to comment on Jim's question about addressing reflexivity?

[11:34] JimDisbrow: I would offer that an ontology without proper use of relationships cannot claim to claim "quality".

[11:22] LeoObrst: @Amanda: (re. Amanda's verbal remark questioning some of the metrics and how they relate to "quality") Is your issue about the definition of "quality"? I think the notion of quality will vary between an ontologist and an application user/manager.

[11:25] AmandaVizedom: @Leo, yes, I am asking whether -- and if so, why -- these characteristics are intrinsic aspects of *quality*. It could be that they are intrinsic metrics, but the relevance to quality depends on extrinsic factors.

[11:25] AmandaVizedom: @Samir, I think that nails it. Thank you. (re. Samir's verbal response.)

[11:25] SamirTartir: @Amanda: Thank you.

[11:25] MariaPovedaVillalon: In addition there is a temporal aspect on that, one class can have few instances today, but will populated later

[11:28] AmandaVizedom: @Leo and @Samir: My reason for wanting the relationship addressed may also be a reason that some reviewers objected: there is a lot of history of these being treated as quality metrics, without any obvious reason. @Samir, if that's true, then making clear the relationship you see, as you articulated it, might well satisfy those critics.

[11:29] DougFoxvog: @Maria & @Mari: if one has several local ontologies, one that includes the other, can the combined ontologies be analyzed together?

[11:29] MariaPovedaVillalon: @doug you can either make them available online so that the owl:imports can be resolved or gather them in one file and paste it into OOPS! textbox

[11:31] LeoObrst: Behind some of this discussion is the presupposition that evaluation is only about quality.

[11:31] PeterYim: @Leo - is that presupposition proper (in the context of this summit) or not?

[11:32] AmandaVizedom: Suggestion: So, we see the likelihood that there are (many, I'd say) suitability is *extrinsic* (at least partially).
 * intrinsic* characteristics of an ontology such the relevance of each characteristics to quality /

[11:33] SteveRay: @Amanda: I agree with you.

[11:34] LeoObrst: @Peter: I think comparison of ontologies is an important issue for ontology evaluation, and one person's notion of "quality" may vary from another person's, so comparing different metrics and allowing weighting of various metrics may be useful.

[11:34] SamirTartir: @Amanda: You mean intrinsic-extrinsic links? That's a good idea.

[11:37] AmandaVizedom: @Leo, that may be so. Or it may simply be that we want/need to clarify the relationship. Some evaluations (or evaluation tools) may be designed to rank ontologies by quality without further information. Those, IMHO, are misguided. What is more promising, IMHO, is a framework/toolkit with the capability of evaluating many characteristics, perhaps neutrally to their relevance to quality in specific cases. It could be up to the user to select which characteristics they care about. Or, in my fantasy system (such as that which JoanneLuciano and other have proposed), a tool in which the use case could be described and ontologies evaluated according to relevant metrics.

[11:37] TerryLongstreth: @Leo - any metacharacteristic may be the basis for a quality judgement, depending on what's important to the user community. Size for example may be impactful in determining what systems resources will need procurement actions.'

[11:38] TorstenHahmann: Some addition to the example that Fabian used in his remark (depth may be useful only in specific contexts): the context also determines how to measure depth. There are dozens of ways one could measure depth, for example: average, shallowest, deepest, relative to breadth, standard deviation, etc. Which of those metrics properly measures the intended quality (a quality "specificity", for example)? Something to explore in the future.

[11:40] DougFoxvog: @Torsten: that would depend upon the task. It might be interesting to define desired features of ontologies & KBs using some of the metrics that have been described.

[11:37] PeterYim: great session ... thanks everyone!

[11:37] SamirTartir: Thank you all. Very interesting, looking forward to more discussions.

[11:37] JimDisbrow: thanks and bye

[11:37] MariCarmenSuarezFigueroa: Thank you very much for your comments and suggestions.

[11:37] MariCarmenSuarezFigueroa: Bye!!

[11:37] AsuncionGomezPerez: bye

[11:37] LeoObrst: Thanks all!

[11:37] MohammadAqtash: Thanks All Bye!

[11:37] MariaPovedaVillalon: Thank you for your comments :-) bye

[11:37] JesualdoTomasFernandezBreis: Thanks and bye!!

[11:38] DougFoxvog: This was a very good session! Bye!

[11:38] PeterYim: join us again, same time next week, for OntologySummit2013 session-04: "Building Ontologies to Meet Evaluation Criteria - I" - Co-chairs: MatthewWest & MikeBennett - http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_02_07

[11:37] PeterYim: -- session ended: 11:37 am PST --

[11:38] List of attendees: AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, AsuncionGomezPerez, BobSchloss, BobbinTeegarden, BruceBray, IsmailcemBudakArpinar, CarmenChui, ClarePaul, DavidMakovoz, DavidLeal, DennisWisnosky, DougFoxvog, DuaneNickull, FabianNeuhaus, FranLightsom, GeraldRadack, HashemShmaisani, HensonGraves, JamesOdell, JesualdoTomasFernandezBreis, JimDisbrow, JoaoPauloAlmeida, JoelBender, JosephTennis, KenBaclawski, LeoObrst, MariCarmenSuarezFigueroa, MariaPovedaVillalon, MarkFox, MatthewWest, MeganKatsumi, MichaelGruninger, MikeDenny, MikeDean, MohammadAqtash, NathalieAussenacGilles, PeterYim, RamSriram, SamirTartir, SteveRay, TerryLongstreth, ToddSchneider, TorstenHahmann, TrishWhetzel, YuvaTarunVarmaDatla, vnc2

-- end of in-session chat-transcript --


 * Further Question & Remarks - please post them to the [ ontology-summit ] listserv
 * all subscribers to the previous summit discussion, and all who responded to today's call will automatically be subscribed to the [ ontology-summit ] listserv
 * if you are already subscribed, post to 
 * (if you are not yet subscribed) you may subscribe yourself to the [ ontology-summit ] listserv, by sending a blank email to  from your subscribing email address, and then follow the instructions you receive back from the mailing list system.
 * (in case you aren't already a member) you may also want to join the ONTOLOG community and be subscribed to the [ ontolog-forum ] listserv, when general ontology-related topics (not specific to this year's Summit theme) are discussed. Please refer to Ontolog membership details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
 * kindly email  if you have any question.

Additional Resources:

 * Homepage of OntologySummit2013
 * Proceedings from the Ontology Summit 2013 Launch Event (2013.01.17) - ConferenceCall_2013_01_17
 * Proceedings from earlier sessions - under: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013#nid3IZA
 * Master Calendar of Events for this Ontology Summit - http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013#nid3J1D
 * [ontology-summit] mailing list archives - http://ontolog.cim3.net/forum/ontology-summit/
 * to subscribe to this discussion list: send a blank message from your subscribing email address to  or visit http://ontolog.cim3.net/mailman/listinfo/ontology-summit/ and subscribe yourself there
 * Ontology Summit 2013 Community Library - http://www.zotero.org/groups/ontologysummit2013
 * Homepage of the Summit - see: OntologySummit

How To Join (while the session is in progress)

 * 1. Dial in with a phone or from skype: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_01_31#nid3LA7
 * 2. Open chat-workspace in a new browser window: http://webconf.soaphub.org/conf/room/summit_20130131
 * 3. Download presentations for each speaker here: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_01_31#nid3L9W
 * or, 3.1 optionally, access our shared-screen vnc server, if you are not behind a corporate firewall

Conference Call Details

 * Date: Thursday, 31-Jan-2013
 * Start Time: 9:30am PST / 12:30pm EST / 6:30pm CET / 17:30 GMT/UTC
 * ref: World Clock
 * Expected Call Duration: ~2.0 hours


 * Dial-in:
 * Phone (US): +1 (206) 402-0100 ... (long distance cost may apply)
 * ... [ backup nbr: (415) 671-4335 ]
 * when prompted enter Conference ID: 141184#
 * Skype: joinconference (i.e. make a skype call to the contact with skypeID="joinconference") ... (generally free-of-charge, when connecting from your computer)
 * when prompted enter Conference ID: 141184#
 * Unfamiliar with how to do this on Skype? ...
 * Add the contact "joinconference" to your skype contact list first. To participate in the teleconference, make a skype call to "joinconference", then open the dial pad (see platform-specific instructions below) and enter the Conference ID: 141184# when prompted.
 * Can't find Skype Dial pad? ...
 * for Windows Skype users: Can't find Skype Dial pad? ... it's under the "Call" dropdown menu as "Show Dial pad"
 * for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later; or on the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. ... (ref.)


 * Shared-screen support (VNC session), if applicable, will be started 5 minutes before the call at: http://vnc2.cim3.net:5800/
 * view-only password: "ontolog"
 * if you plan to be logging into this shared-screen option (which the speaker may be navigating), and you are not familiar with the process, please try to call in 5 minutes before the start of the session so that we can work out the connection logistics. Help on this will generally not be available once the presentation starts.
 * people behind corporate firewalls may have difficulty accessing this. If that is the case, please download the slides above (where applicable) and running them locally. The speaker(s) will prompt you to advance the slides during the talk.


 * In-session chat-room url: http://webconf.soaphub.org/conf/room/summit_20130131
 * instructions: once you got access to the page, click on the "settings" button, and identify yourself (by modifying the Name field from "anonymous" to your real name, like "JaneDoe").
 * You can indicate that you want to ask a question verbally by clicking on the "hand" button, and wait for the moderator to call on you; or, type and send your question into the chat window at the bottom of the screen.
 * thanks to the soaphub.org folks, one can now use a jabber/xmpp client (e.g. gtalk) to join this chatroom. Just add the room as a buddy - (in our case here) summit_20130131@soaphub.org ... Handy for mobile devices!


 * Discussions and Q & A:
 * Nominally, when a presentation is in progress, the moderator will mute everyone, except for the speaker.
 * To un-mute, press "*7" ... To mute, press "*6" (please mute your phone, especially if you are in a noisy surrounding, or if you are introducing noise, echoes, etc. into the conference line.)
 * we will usually save all questions and discussions till after all presentations are through. You are encouraged to jot down questions onto the chat-area in the mean time (that way, they get documented; and you might even get some answers in the interim, through the chat.)
 * During the Q&A / discussion segment (when everyone is muted), If you want to speak or have questions or remarks to make, please raise your hand (virtually) by clicking on the "hand button" (lower right) on the chat session page. You may speak when acknowledged by the session moderator (again, press "*7" on your phone to un-mute). Test your voice and introduce yourself first before proceeding with your remarks, please. (Please remember to click on the "hand button" again (to lower your hand) and press "*6" on your phone to mute yourself after you are done speaking.)


 * Please review our Virtual Session Tips and Ground Rules - see: VirtualSpeakerSessionTips


 * RSVP  to [mailto:peter.yim@cim3.com peter.yim@cim3.com] with your affiliation appreciated, ... or simply just by adding yourself to the "Expected Attendee" list below (if you are a member of the community already.)


 * This session, like all other Ontolog events, is open to the public. Information relating to this session is shared on this wiki page: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_01_31


 * Please note that this session may be recorded, and if so, the audio archive is expected to be made available as open content, along with the proceedings of the call to our community membership and the public at-large under our prevailing open IPR policy.

Attendees

 * Attended:
 * SteveRay (co-chair)
 * LeoObrst (co-chair)
 * MariaPovedaVillalon
 * MariCarmenSuarezFigueroa
 * AsuncionGomezPerez
 * SamirTartir
 * IsmailcemBudakArpinar
 * JesualdoTomasFernandezBreis
 * AstridDuqueRamos
 * NathalieAussenacGilles
 * MichaelGruninger
 * MatthewWest
 * RamSriram
 * MikeDean
 * AmandaVizedom
 * FabianNeuhaus
 * TerryLongstreth
 * ToddSchneider
 * PeterYim
 * AlejandroMiraAgudelo
 * AnatolyLevenchuk
 * BobSchloss
 * BobbinTeegarden
 * BruceBray
 * CarmenChui
 * ClarePaul
 * DavidMakovoz
 * DavidLeal
 * DennisWisnosky
 * DougFoxvog
 * DuaneNickull
 * FranLightsom
 * GeraldRadack
 * HashemShmaisani
 * HensonGraves
 * JamesOdell
 * JimDisbrow
 * JoaoPauloAlmeida
 * JoelBender
 * JosephTennis
 * KenBaclawski
 * MariaCopeland
 * MarkFox
 * MeganKatsumi
 * MikeDenny
 * MohammadAqtash
 * TorstenHahmann
 * TrishWhetzel
 * YuvaTarunVarmaDatla


 * Expecting:
 * SteveRay (co-chair)
 * LeoObrst (co-chair)
 * MariaPovedaVillalon
 * MariCarmenSuarezFigueroa
 * AsuncionGomezPerez
 * SamirTartir
 * IsmailcemBudakArpinar
 * AmitSheth
 * JesualdoTomasFernandezBreis
 * AstridDuqueRamos
 * RobertStevens
 * NathalieAussenacGilles
 * MichaelGruninger
 * MatthewWest
 * RamSriram
 * MikeDean
 * AmandaVizedom
 * FabianNeuhaus
 * TerryLongstreth
 * ToddSchneider
 * MikeBennett
 * PeterYim
 * ScottHills
 * JosephTennis (will need to leave 30 minutes early)
 * MeganKatsumi
 * PavithraKenjige
 * CarmenChui
 * TorstenHahmann
 * HansPolzer
 * JoaoPauloAlmeida
 * JimDisbrow
 * FranLightsom
 * DougFoxvog
 * MikeDenny
 * BobbinTeegarden
 * BobSchloss (first 30 minutes only)
 * please add yourself to the list if you are a member of the Ontolog or OntologySummit community, or, rsvp to  with your affiliation.
 * please add yourself to the list if you are a member of the Ontolog or OntologySummit community, or, rsvp to  with your affiliation.


 * Regrets:
 * MarcelaVegetti
 * JoanneLuciano
 * DeborahNichols