Moving Up to the Cloud, a panel lecture hosted by the VCU Libraries

“Sky symphony” by Kevin Dooley

“Educational Utility Computing: Perspectives on .edu and the Cloud”
Mark Ryland, Chief Solutions Architect at Amazon Web Services

AWS has been a part of revolutionizing the start-up industries (i.e. Instagram, Pinterest) because they don’t have the cost of building server infrastructures in-house. Cloud computing in the AWS sense is utility computing — pay for what you use, easy to scale up and down, and local control of how your products work. In the traditional world, you have to pay for the capacity to meet your peak demand, but in the cloud computing world, you can level up and down based on what is needed at that moment.

Economies, efficiencies of scale in many ways. Some obvious: storage, computing, and networking equipment supply change; internet connectivity and electric power; and data center sitting, redundancy, etc. Less obvious: security and compliance best practices; datacenter internal innovations in networking, power, etc.

AWS and .EDU: EdX, Coursera, Texas Digital Library, Berkeley AMP Lab, Harvard Medical, University of Phoenix, and an increasing number of university/school public-facing websites.

Expects that we are heading toward cloud computing utilities to function much like the electric grid — just plug in and use it.


“Libraries in Transition”
Marshall Breeding, library systems expert

We’ve already seen the shift of print to electronic in academic journals, and we’re heading that way with books. Our users are changing in the way they expect interactions with libraries to be, and the library as space is evolving to meet that, along with library systems.

Web-based computing is better than client/server computing. We expect social computing to be integrated into the core infrastructure of a service, rather than add-ons and afterthoughts. Systems need to be flexible for all kinds of devices, not just particular types of desktops. Metadata needs to evolve from record-by-record creation to bulk management wherever possible. MARC is going to die, and die soon.

How are we going to help our researchers manage data? We need the infrastructure to help us with that as well. Semantic web — what systems will support it?

Cooperation and consolidation of library consortia; state-wide implementations of SaaS library systems. Our current legacy ILS are holding libraries back from being able to move forward and provide the services our users want and need.

A true cloud computing system comes with web-based interfaces, externally hosted, subscription OR utility pricing, highly abstracted computing model, provisioned on demand, scaled according to variable needs, elastic.


“Moving Up to the Cloud”
Mark Triest, President of Ex Libris North America

Currently, libraries are working with several different systems (ILS, ERMS, DRs, etc.), duplicating data and workflows, and not always very accurately or efficiently, but it was the only solution for handling different kinds of data and needs. Ex Libris started in 2007 to change this, beginning with conversations with librarians. Their solution is a single system with unified data and workflows.

They are working to lower the total cost of ownership by reducing IT needs, minimize administration time, and add new services to increase productivity. Right now there are 120+ institutions world-wide who are in the process of or have gone live with Alma.

Automated workflows allow staff to focus on the exceptions and reduce the steps involved.

Descriptive analytics are built into the system, with plans for predictive analytics to be incorporated in the future.

Future: collaborative collection development tools, like joint licensing and consortial ebook programs; infrastructure for ad-hoc collaboration


“Cloud Computing and Academic Libraries: Promise and Risk”
John Ulmschneider, Dean of Libraries at VCU

When they first looked at Alma, they had two motivations and two concerns. They were not planning or thinking about it until they were approached to join the early adopters. All academic libraries today are seeking to discover and exploit new efficiencies. The growth of cloud-resident systems and data requires academic libraries to reinvigorate their focus on core mission. Cloud-resident systems are creating massive change throughout out institutions. Managing and exploiting pervasive change is a serious challenge. Also, we need to deal with security and durability of data.

Cloud solutions shift resources from supporting infrastructure to supporting innovation.

Efficiencies are not just nice things, they are absolutely necessary for academic libraries. We are obligated to upend long-held practice, if in doing so we gain assets for practice essential to our mission. We must focus recovered assets on the core library mission.

Agility is the new stability.

Libraries must push technology forward in areas that advance their core mission. Infuse technology evolution for libraries with the values needs of libraries. Libraries must invest assets as developers, development partners, and early adopters. Insist on discovery and management tools that are agnostic regarding data sources.

Managing the change process is daunting.. but we’re already well down the road. It’s not entirely new, but it does involve a change in culture to create a pervasive institutional agility for all staff.

IL 2012: Discovery Systems

Space Shuttle Discovery Landing At Washington DC
“Space Shuttle Discovery Landing At Washington DC” by Glyn Lowe

Speaker: Bob Fernekes

The Gang of Four: Google, Apple, Amazon, & Facebook

Google tends to acquire companies to grow the capabilities of it. We all know about Apple. Amazon sells more ebooks than print books now. Facebook is… yeah. That.

And then we jump to selecting a discovery service. You would do that in order to make the best use of the licensed content. This guy’s library did a soft launch in the past year of the discovery service they chose, and it’s had an impact on the instruction and tools (i.e. search boxes) he uses.

And I kind of lost track of what he was talking about, in part because he jumped from one thing to the next, without much of a transition or connection. I think there was something about usability studies after they implemented it, although they seemed to focus on more than just the discovery service.

Speaker: Alison Steinberg Gurganus

Why choose a discovery system? You probably already know. Students lack search skills, but they know how to search, so we need to give them something that will help them navigate the proprietary stuff we offer out on the web.

The problem with the discovery systems is that they are very proprietary. They don’t quite play fairly or nicely with competitor’s content yet.

Our users need to be able to evaluate, but they also need to find the stuff in the first place. A great discovery service should be self-explanatory, but we don’t have that yet.

We have students who understand Google, which connects them to all the information and media they want. We need something like that for our library resources.

When they were implementing the discovery tool, they wanted to make incremental changes to the website to direct users to it. They went from two columns, with the left column being text links to categories of library resources and services, to three columns, with the discover search box in the middle column.

When they were customizing the look of the discovery search results, they changed the titles of items to red (from blue). She notes that users tend to ignore the outside columns because that’s where Google puts advertisements, so they are looking at ways to make that information more visible.

I also get the impression that she doesn’t really understand how a discovery service works or what it’s supposed to do.

Speaker: Athena Hoeppner

Hypothesis: discovery includes sufficient content of high enough quality, with full text, and …. (didn’t type fast enough).

Looked at final papers from a PhD level course (34), specifically the methodology section and bibliography. Searched for each item in the discovery search as well as one general aggregator database and two subject-specific databases. The works cited were predominately articles, with a significant number of web sources that were not available through library resources. She was able to find more citations in the discovery search than in Google Scholar or any of the other library databases.

Clearly the discovery search was sufficient for finding the content they needed. Then they used a satisfaction survey of the same students that covered familiarity and frequency of use for the subject indexes, discovery search, and Google Scholar. Ultimately, it came down that the students were satisfied and happy with the subject indexes, and too few respondents to get a sense of satisfaction with the discovery search or Google Scholar.

Conclusions: Students are unfamiliar with the discovery system, but it could support their research needs. However, we don’t know if they can find the things they are looking for in it (search skills), nor do we know if they will ultimately be happy with it.

ER&L 2012 – Between Physical and Digital: Understanding Cross-Channel User Experiences

UX Brighton 2011 - Andrea Resmini
photo by Katariina Järvinen

speaker: Andrea Resmini

He starts with a brief description of the movie The Name of the Rose, which is a bit of a medieval murder mystery involving a monastery library. The “library” is actually a labyrinth, but only in the movie. (The book is a little different.)

The letters on the arches represent the names of the places in the world, and are placed in the library where they would be in the world as it relates to Europe. They didn’t exactly replicate the world, but they ordered it like good librarians.

If you don’t understand the organizational system, it’s just a labyrinth. The movie had to change this because it wouldn’t work to have room after room of books covering the walls. We have to see the labyrinth to be able to participate in the experience, which can be different depending on the medium (book or movie).

Before computers, we relied on experts (people), books, and mentors to learn. With computers, we have access to all of them, at any time. We are constantly connected (if we choose) to streams of data, and the access points are more and more portable.

“Cyberspace is not a place you go to but rather a layer tightly integrated into the world around us.” –Institute for the Future

This is not the future. It’s here now. Facebook, Twitter, Foursquare… our phones and mobile devices connect us.

Think about how you might send a message? Email, text, handwritten, smoke signals, ouija… ti’s the same task, but with many different mediums.

What if someone is looking for a book? They could go to the circ desk, but that’s becoming less common. They could go to a virtual bookshelf for the library. Or they could go to a competitor like Amazon. They could do this on a mobile phone. Or they could just start looking on the shelves themselves, whether they understand the classification/organization or not. The only thing that matters is the book. They don’t want to fight with mobile interfaces, search results in the millions, or creepy library stacks. They just want the book, when they want it, and how they want it.

The library is a channel, as is the labeling, circ desk, website, mobile interface, etc. Unfortunately, they don’t work together. We have silos of channels, not just silos of information.

Think about a bank. You can talk to the call center employee — they can’t help you if it’s not a part of their scripted routines. You can’t start an online process and finish it in a physical space (i.e. online banking then local branch).

Entertainment now uses many channels to reach consumers. If you really want to understand the second and third Matrix movies, you have to be familiar with the accessory channels of information (comic books, video games, etc.). In cross-channel experiences, users constantly move between channels, and will not stay in any single one of them from start to finish.

More companies, like clothing stores, are breaking down the barriers to flow between their physical and virtual stores. You can shop on line and return items to the physical store, for example.

Manifesto:

  1. Information architectures are becoming open ecologies: no artifacts stand alone — they are all apart of the user experience
  2. users are becoming intermediaries: participants in these ecosystems actively produce and re-mediate content and meaning
  3. static becomes dynamic: ecologies are perpetually unfinished, always changing, always open to further refinement and manipulation
  4. dynamic becomes hybrid: the boundaries separating media, channels, and genres get thinner
  5. horizontal prevails over vertical: intermediaries push for spontaneity, ephemeral structures of meaning and constant change
  6. products are becoming experiences: focus shifts from how to design single items to how to design experiences spanning multiple steps
  7. experiences become cross-channel experiences: experiences bridge multiple connected media, devices and environments into ubiquitous ecologies

VLACRL Spring 2011: Building an eReaders Collection at Duke University Libraries

They started lending ereaders because they wanted to provide a way for users to interact with new and emerging technologies.

Speaker: Nancy Gibbs

They started lending ereaders because they wanted to provide a way for users to interact with new and emerging technologies. The collection focus is on high circulation popular reading titles, and they do add patron requests. Recently, they added all of the Duke University Press titles, per the request of the university press. (Incidentally, not all of the Duke UP titles are available in Kindle format because Amazon won’t allow them to sell a book in Kindle format until it has sold 50 print copies.)

They marketed their ereader program through word of mouth, the library website, the student paper, and the communications office. The communications press release was picked up by the local newspaper. They also created a YouTube video explaining how to reserve/check-out the ereaders, and gave presentations to the teaching & learning technologists and faculty.

For the sake of consistency and availability of titles, they purchase one copy of a title for every pod of six Kindle ereaders. Amazon allows you to load and view a Kindle book on up to six devices, which is how they arrived at that number. For the Nooks, they can have a book loaded on apparently an unlimited number of devices, so they purchase only one copy of a title from Barnes & Noble. They try to have the same titles on both the Kindles and the Nooks, but not every title available for purchase on the Kindle is also available on the Nook. Each of the books purchased is cataloged individually, with the location as the device it is on, and they will appear to be checked out when the device is checked out.

When they first purchased the devices and were figuring out the local workflow of purchasing and loading the content, the tech services department (acquisitions, cataloging, etc.) were given the devices to experiment with them. In part, this was to sort out any kinks in workflow that they may discover, but also it was because these folks don’t often get the chance to play with new technology in the library as their public service counterparts do. Gibbs recommends that libraries purchase insurance options for the devices, because things can happen.

One of the frustrations with commercial ereader options like the Kindle and Nook is that they are geared towards individual users and not library use. So, unlike other ebook providers and platforms, they do not give the library any usage data regarding the books used, which can make collection development in these areas somewhat challenging. However, given that their scope is popular reading material and that they take patron requests, this is not as much of an issue as it could be.

Side note: Gibbs pointed out that ebook readers are still not yet greener than print books, mostly because of the toxicity of the materials and the amount of resources that go into producing them. EcoLibris has a great resource page with more information about this.

library lending with the Kindle

kindle with newspaper
Amazon Kindle

I’m sure by now that you’ve heard the Amazon announcement that they will be offering a service to allow libraries to lend books to Kindle users. Well, the thing that got this academic librarian excited is this line from the press release: “If a Kindle book is checked out again or that book is purchased from Amazon, all of a customer’s annotations and bookmarks will be preserved.”

One of the common complaints we received in our pilot programs using Kindles in the classroom was that because the students had to return the devices, they couldn’t keep the notes they had made in the texts. Of course, even with this model they won’t be able to access their notes without checking out the book again, but at least it’ll be an option for them.

Of course, there is a down side to this announcement — the lending will be facilitated by OverDrive.  Unless you’ve been a library news hermit for the past few years, you’ve heard the complaints (and very few praises) about the OverDrive platform, and the struggles of librarians and users in getting the materials checked out and downloaded to devices. I hope that because Amazon will be relying on their fantastic Whispersync technology to retain notes and bookmarks,it will just as easy to check out and download the Kindle books through OverDrive.

ER&L: Library Renewal

Speaker: Michael Porter

Libraries are content combined with community. Electronic content access is making it more challenging for libraries to accomplish their missions.

It’s easy to complain but hard to do. Sadly, we tend to complain more than doing. If we get a reputation for being negative, that will be detrimental. That doesn’t mean we should be Sally Sunshine, but we need to approach things with a more positive attitude to make change happen.

Libraries have an identity problem. We are tied to content (ie books). 95% of people in poverty have cable television. They can’t afford it, but they want it, so they get it. Likewise, mobile access to content is moving to ubiquitous.

Our identity needs to be moved back to content. We need to circ electronic content better than Netflix, Amazon, iTunes, etc.

Electronic content distribution is a complicated issue. Vendors in our market don’t have the same kind of revenue as companies like Apple. We aren’t getting the best people or solutions — we’re getting the good enough, if we’re lucky.

Could libraries became the distribution hub for media and other electronic content?

WordCamp Richmond: Exploiting Your Niche – Making Money with Affiliate Marketing

presenter: Robert Sterling

Affiliate marketing is a practice of rewarding an affiliate for directing customers to the brand/seller that then results in a sale.

“If you’re good at something, never do it for free.” If you have a blog that’s interesting and people are coming to you, you’re doing something wrong if you’re not making money off of it.

Shawn Casey came up with a list of hot niches for affiliate marketing, but that’s not how you find what will work for you. Successful niches tend to be what you already have a passion for and where it intersects with affiliate markets. Enthusiasm provokes a positive response. Enthusiasm sells. People who are phoning it in don’t come across the same and won’t develop a loyal following.

Direct traffic, don’t distract from it. Minimize the number of IAB format ads – people don’t see them anymore. Maximize your message in the hot spots – remember the Google heat map. Use forceful anchor text like “click here” to direct users to the affiliate merchant’s site. Clicks on images should move the user towards a sale.

Every third or fourth blog post should be revenue-generating. If you do it with every post, people will assume it’s a splog. Instapundit is a good example of how to do a link post that directs users to relevant content from affiliate merchants. Affiliate datafeeds can be pulled in using several WP plugins. If your IAB format ads aren’t performing from day one, they never will.

Plugins (premium): PopShops works with a number of vendors. phpBay/phpZon works with eBay and Amazon, respectively. They’re not big revenue sources, but okay for side money.

Use magazine themes that let you prioritize revenue-generating content. Always have a left-sidebar and search box, because people are more comfortable with that navigation.

Plugins (free): W3 Total Cache (complicated, buggy, but results in fast sites, which Google loves), Regenerate Thumbnails, Ad-minister, WordPress Mobile, and others mentioned in previous sessions. Note: if you change themes, make sure you go back and check old posts. You want them to look good for the people who find them via search engines.

Forum marketing can be effective. Be a genuine participant, make yourself useful, and link back to your site only occasionally. Make sure you optimize your profile and use the FeedBurner headline animator.

Mashups are where you can find underserved niches (i.e. garden tools used as interior decorations). Use Google’s keyword tools to see if there is a demand and who may be your competition. Check for potential affiliates on several networks (ClickBank, ShareASale, Pepperjam, Commission Junction, and other niche-appropriate networks). Look for low conversion rates, and if the commission rate is less than 20%, don’t bother.

Pay for performance (PPP) advertising is likely to replace traditional retail sales. Don’t get comfortable – it’s easy for people to copy what works well for you, and likewise you can steal from your competition.

Questions:

What’s a good percentage to shoot for? 50% is great, but not many do that. Above 25% is a good payout. Unless the payout is higher, avoid the high conversion rate affiliate programs. Look for steady affiliate marketing campaigns from companies that look like they’re going to be sticking around.

What about Google or Technorati ads? The payouts have gone down. People don’t see them, and they (Google) aren’t transparent enough.

How do you do this not anonymously and maintain integrity in the eyes of your readers? One way to do it is a comparison post. Look at two comparable products, list their features against each other.

NASIG 2010: Publishing 2.0: How the Internet Changes Publications in Society

Presenter: Kent Anderson, JBJS, Inc

Medicine 0.1: in dealing with the influenza outbreak of 1837, a physician administered leeches to the chest, James’s powder, and mucilaginous drinks, and it worked (much like take two aspirin and call in the morning). All of this was written up in a medical journal as a way to share information with peers. Journals have been the primary source of communicating scholarship, but what the journal is has become more abstract with the addition of non-text content and metadata. Add in indexes and other portals to access the information, and readers have changed the way they access and share information in journals. “Non-linear” access of information is increasing exponentially.

Even as technology made publishing easier and more widespread, it was still producers delivering content to consumers. But, with the advent of Web 2.0 tools, consumers now have tools that in many cases are more nimble and accessible than the communication tools that producers are using.

Web 1.0 was a destination. Documents simply moved to a new home, and “going online” was a process separate from anything else you did. However, as broadband access increases, the web becomes more pervasive and less a destination. The web becomes a platform that brings people, not documents, online to share information, consume information, and use it like any other tool.

Heterarchy: a system of organization replete with overlap, multiplicity, mixed ascendandacy and/or divergent but coextistent patterns of relation

Apomediation: mediation by agents not interposed between users and resources, who stand by to guide a consumer to high quality information without a role in the acquisition of the resources (i.e. Amazon product reviewers)

NEJM uses terms by users to add related searches to article search results. They also bump popular articles from searches up in the results as more people click on them. These tools improved their search results and reputation, all by using the people power of experts. In addition, they created a series of “results in” publications that highlight the popular articles.

It took a little over a year to get to a million Twitter authors, and about 600 years to get to the same number of book authors. And, these are literate, savvy users. Twitter & Facebook count for 1.45 million views of the New York Times (and this is a number from several years ago) — imagine what it can do for your scholarly publication. Oh, and NYT has a social media editor now.

Blogs are growing four times as fast as traditional media. The top ten media sites include blogs and the traditional media sources use blogs now as well. Blogs can be diverse or narrow, their coverage varies (and does not have to be immediate), they are verifiably accurate, and they are interactive. Blogs level that media playing field, in part by watching the watchdogs. Blogs tend to investigate more than the mainstream media.

It took AOL five times as long to get to twenty million users than it did for the iPhone. Consumers are increasingly adding “toys” to their collection of ways to get to digital/online content. When the NEJM went on the Kindle, more than just physicians subscribed. Getting content into easy to access places and on the “toys” that consumers use will increase your reach.

Print digests are struggling because they teeter on the brink of the daily divide. Why wait for the news to get stale, collected, and delivered a week/month/quarter/year later? People are transforming. Our audiences don’t think of information as analogue, delayed, isolated, tethered, etc. It has to evolve to something digital, immediate, integrated, and mobile.

From the Q&A session:

The article container will be here for a long time. Academics use the HTML version of the article, but the PDF (static) version is their security blanket and archival copy.

Where does the library as source of funds when the focus is more on the end users? Publishers are looking for other sources of income as library budgets are decreasing (i.e. Kindle, product differentiation, etc.). They are looking to other purchasing centers at institutions.

How do publishers establish the cost of these 2.0 products? It’s essentially what the market will bear, with some adjustments. Sustainability is a grim perspective. Flourishing is much more positive, and not necessarily any less realistic. Equity is not a concept that comes into pricing.

The people who bring the tremendous flow of information under control (i.e. offer filters) will be successful. One of our tasks is to make filters to help our users manage the flow of information.

CIL 2010: The Power in Your Browser – LibX & Zotero

Speaker: Krista Godfrey

She isn’t going to show how to create LibX or Zotero access, but rather how to use them to create life-long learners. Rather than teaching students how to use proprietary tools like Refworks, teaching them tools they can use after graduation will help support their continued research needs.

LibX works in IE and Firefox. They are working on a Chrome version as well. It fits into the search and discovery modules in the research cycle. The toolbar connects to the library catalog and other tools, and right-click menu search options are available on any webpage.  It will also embed icons in places like Amazon that will link to catalog searches, and any page with a document identifier (DOI, ISSN) will now present that identifier as a link to the catalog search.

Zotero is only in Firefox, unfortunately. It’s a records management tool that allows you to collect, manage, cite, and share, which fill in the rest of the modules in the research cycle. It will collect anything, archive anything, and store any attached documents. You can add notes, tags, and enhance the metadata. The citation process works in Word, Open Office, and Google Docs, with a program similar to Write-N-Cite that can be done by dragging and dropping the citation where you want it to go.

One of the down-sides to Zotero when it first came out was that it lived only in one browser on one machine, but the new version comes with server space that you can sync your data to, which allows you to access your data on other browsers/machines. You can create groups and share documents within them, which would be great for a class project.

Why aren’t we teaching Zotero/LibX more? Well, partially because we’ve spent money on other stuff, and we tend to push those more. Also, we might be worried that if we give our users tools to access our content without going through our doors, they may never come back. But, it’s about creating life-long learners, and they won’t be coming through our doors when they graduate. So, we need to teach them tools like these.

ER&L 2010: Adventures at the Article Level

Speaker: Jamene Brooks-Kieffer

Article level, for those familiar with link resolvers, means the best link type to give to users. The article is the object of pursuit, and the library and the user collaborate on identifying it, locating it, and acquiring it.

In 1980, the only good article-level identification was the Medline ID. Users would need to go through a qualified Medline search to track down relevant articles, and the library would need the article level identifier to make a fast request from another library. Today, the user can search Medline on their own; use the OpenURL linking to get to the full text, print, or ILL request; and obtain the article from the source or ILL. Unlike in 1980, the user no longer needs to find the journal first to get to the article. Also, the librarian’s role is more in providing relevant metadata maintenance to give the user the tools to locate the articles themselves.

In thirty years, the library has moved from being a partner with the user in pursuit of the article to being the magician behind the curtain. Our magic is made possible by the technology we know but that our users do not know.

Unique identifiers solve the problem of making sure that you are retrieving the correct article. CrossRef can link to specific instances of items, but not necessarily the one the user has access to. The link resolver will use that DOI to find other instances of the article available to users of the library. Easy user authentication at the point of need is the final key to implementing article-level services.

One of the library’s biggest roles is facilitating access. It’s not as simple as setting up a link resolver – it must be maintained or the system will break down. Also, document delivery service provides an opportunity to generate goodwill between libraries and users. The next step is supporting the users preferred interface, through tools like LibX, Papers, Google Scholar link resolver integration, and mobile devices. The latter is the most difficult because much of the content is coming from outside service providers and the institutional support for developing applications or web interfaces.

We also need to consider how we deliver the articles users need. We need to evolve our acquisitions process. We need to be ready for article-level usage data, so we need to stop thinking about it as a single-institutional data problem. Aggregated data will help spot trends. Perhaps we could look at the ebook pay-as-you-use model for article-level acquisitions as well?

PIRUS & PIRUS 2 are projects to develop COUNTER-compliant article usage data for all article-hosting entities (both traditional publishers and institutional repositories). Projects like MESUR will inform these kinds of ventures.

Libraries need to be working on recommendation services. Amazon and Netflix are not flukes. Demand, adopt, and promote recommendation tools like bX or LibraryThing for Libraries.

Users are going beyond locating and acquiring the article to storing, discussing, and synthesizing the information. The library could facilitate that. We need something that lets the user connect with others, store articles, and review recommendations that the system provides. We have the technology (magic) to make it available right now: data storage, cloud applications, targeted recommendations, social networks, and pay-per-download.

How do we get there? Cover the basics of identify>locate>acquire. Demand tools that offer services beyond that, or sponsor the creation of desired tools and services. We also need to stay informed of relevant standards and recommendations.

Publishers will need to be a part of this conversation as well, of course. They need to develop models that allow us to retain access to purchased articles. If we are buying on the article level, what incentive is there to have a journal in the first place?

For tenure and promotion purposes, we need to start looking more at the impact factor of the article, not so much the journal-level impact. PLOS provides individual article metrics.