Technology Assessment 101 - Information Gathering

Time put towards planning and performing research at the beginning of the technology assessment process is time well spent.  Decisions made later in the assessment process will largely be directed by the work done in these critical first steps.  There are really two primary areas to focus on in these initial efforts – determining what your organization needs, and determining what the current market can offer your telehealth program.

Needs Assessment

Organizations may find themselves starting the process of technology assessment with little more than a high-level goal or position statement.  More likely, however, is an organization with a researched need, documented disparity in available services, or established telehealth program that has been so successful that other clinical services are requesting to join the telehealth network.  Regardless of the level of information gathered, work will need to be done to compile information in a manner that is useful to those who will be performing the technology assessment.  It is important to pull together people from relevant areas of your organization at this early stage of the process, so that clinicians, technical experts, and organizational administrators can provide input.

Forming a Vision

The need to perform a technology assessment does not spontaneously develop; there is a need – often clinical or financial – that can only practically be met through some variety of telehealth service.  While this initial need is often fairly instructive, work should be done to clarify precisely what is desired in the implementation of a new telehealth service or program.

Position Statements

A position statement is one approach to developing what might otherwise be a high-level or unclear goal.  Position statements often maintain this high-level view, but help explain why your organization is starting an endeavor and what the end goal is.  Position statements typically address who is being served, why they will be interested in your services, what your service will offer, why that is important, and how it is different from other organizations or other ways of providing care. 

For example, if your organization plans to start a teledermatology program, it may be useful to further define what exactly is intended.  Your position statement may be “we will provide teledermatology to patients in remote locations that lack access to broadband connectivity, who live in underserved locations that lack dermatology specialty services.  We will provide 20 hours of specialty services a week to our partner organizations, who otherwise would not have access to dermatology services without incurring a significant cost due to travel.” 

The people who started the initiative may understand who the program will serve, and why it is important.  A statement like this can give people who join the project later on a clear vision for this endeavor, and helps provide some guidance to future decisions.  This process, which may appear to be too high-level to be of much use, will be helpful later on.  The above statement clarifies that there are connectivity problems, that participating clinics are partnered with your organization, and that you are focusing on places that do not already have access to dermatology services.   

Establishing Scope

While the position statement may help provide some general information about where the project intends to go, additional work must be done to clearly define what this project will cover.  Specifically, it can be very useful to establish user profiles and use cases, and to perform a SWOT analysis.

User Profiles

The process of creating user profiles can serve to define a set of needs that will need to be considered during later technology assessment activities.  These profiles can be as detailed or as broad as an organization wants; the level of detail that is most useful to an organization may vary.  At a very high level, a profile might simply specify the titles of intended users.  More detailed descriptions may include known concerns within a user base, such as discomfort with technology, specific workflows, prejudices and biases, etc.  Regardless of the level of detail that your organization chooses to provide with each user profile, it is important to spend time thinking about who will be using this technology, and how they will interact with it.

Use Cases

As with user profiles, use cases can be as high-level or as detailed as necessary.  Whereas user profiles look at who will use the system, use cases look at how they will use it.  Your organization may want to consider the following when creating use cases:

  • Are there particular physical constraints on the technology, such as the need to use devices with one hand, the need to frequently move equipment between rooms and other physical locations?
  • Does the equipment need to interoperate with existing technology or systems?
  • How will the technology fit into existing infrastructure and workflows?
  • How will different users (possibly specified in user profiles) interact with the system?  Will people use the devices in different ways, depending on their role?

Requirements Gathering

The work done creating user profiles and use cases can lead to a set of clear requirements for the devices.  Create a list of features that the devices must have, as well as a list of features that are desired and those that are either unnecessary or detrimental to meeting the needs of your user profiles and use cases.  This initial list of requirements that are gathered will likely lead to the development of additional requirements.  Discuss with others in the evaluation team:

  • Why is that feature needed?  Is it really needed, or is it optional?
  • How will that requirement impact users?
  • Have the needs of all people who will interact with the device been considered – clinicians, patients, technical staff, maintenance staff, cleaning staff?

SWOT Analysis

SWOT (Strength, Weakness, Opportunity, Threat) assessments will require input from many different areas of your organization.  This process, while not absolutely critical to a successful device evaluation, can help establish a list of resources and concerns for the assessment process.  Again, the importance of an interdisciplinary team cannot be emphasized enough at this point.  This assessment looks at strengths (things that your organization does well), weaknesses (things that your organization needs to improve), opportunities (external factors that your organization should act on or could benefit from), and threats (external factors that pose a risk to your organization).

There are many resources in print and online that look at how to properly perform a SWOT analysis.  Using the materials that are available and any organizational skills in this practice, performing a SWOT analysis can help further refine what you are trying to do, and why.

This process may need to be repeated after additional information is acquired.  Should, for example, a network engineer join the assessment team after the initial SWOT process is performed, it can be useful to ask the new team member to think through what strengths, weaknesses, opportunities, and threats may exist in the area of their expertise.

Market Review

Ideally, you organization will have a firm grasp of what the general requirements are at the end of the needs assessment process.  The next step is to look at what is available on the market.  This review should include discussions with vendors and manufacturers, various third-party assessments of the devices, product specifications, and additional research into fundamental principles behind the technology.

Vendors and Manufacturers

Working with the people who build and sell any particular technology can be an interesting balance between gathering information and processing sales pitches.  Many manufacturers are excited to talk about the fundamental principles being their technology.  Having a person in your organization who can clearly communicate with a manufacturer’s engineering staff can be useful when gathering information about how technology works.

Discuss the options that exist within a manufacturer’s product line, and see if there are important distinctions between various offerings.  Most manufacturers and vendors will be able to provide a product cut sheet that lists specifications and additional information on how the device works.  Be wary of manufacturer-specific terminology; request that any unclear terms be defined and described so that objective comparisons can be made between specification sheets from different manufacturers.

These initial interactions can be very useful in the information-gathering process.  Clearly explain what you are doing regarding technology assessment, and let the vendor or manufacturer know why you are considering implementing a technology.  The vendor or manufacturer may be able to provide advice about how to test the device, and may point you to the appropriate product or products in their catalog that will meet your needs.

Third-Party Assessments

Depending on the particular technology being reviewed, third party organizations may be able to provide some input as to how the devices work.  Consumer technologies tend to have a variety of options available, such Consumer Reports or other product-specific review sites.  Digital cameras and camcorders are great examples of technologies with extensive third-party reviews.  The problem with some consumer review sites is that they may not consider the clinical application of various devices; digital cameras that make a person look great in front of the Grand Canyon might not produce an image that has colors and lighting balances that make the most clinically-useful photo.

Telehealth-specific review sites are less available than those that address consumer markets.  Talk to other programs that have implemented the technology in the past to see how their experience with a device has worked out.  Also consider communicating with your regional telehealth resource center to see if they have information that might be useful in the market review process.

Product Specifications

The cut sheets and other documentation provided for a device can be useful.  The information may help highlight trends or features that were not considered in the initial requirements gathering process.  Performing side-by-side comparisons between specification sheets can show areas that may need more research.

The TTAC has found it useful to create a product specification comparison table when performing this portion of the market review, as it helps to establish feature parity between devices, or indicate where there might be differences.  The initial comparison document will often change by the end of the review, as new features are discovered and an understanding of the technology is increased.

Technology Overview Documentation

A set of product specifications and list of available products on the market may be sufficient for an organization to move into test planning, but there may still be some utility in further refining this information into a format that is more easily used by those who will be involved in the testing process.  A document that explains how the technology works can be useful in ensuring that testers know what to expect, and can help explore the ties that may exist between a technology, its intended use, and the current state of the market.

Back to Top