The Medicare Prescription Drug Plan Rating System is based on  based on the survey data. Plans are given a number of  "stars" out of a maximum of five (5) stars.

Here is a summary from the Medicare.gov site that gives more of an explanation of the Part D plan rating System:

Member Experience with Drug Plan: This rating shows how the drug plan performed in Medicare’s member satisfaction survey, including drug plan scores in the following areas:

• How often the drug plan provides information or help when members need it.
  
  
• How members rate the drug plan overall.
  
  
• How often members could get prescriptions filled easily using the drug plan.

Drug Plan Customer Service: This category shows how Medicare and members rate the drug plan and how well a drug plan provides customer service.

• Time on Hold When Customer Calls Drug Plan: The data used to measure the performance of the drug plan’s customer service come from call center surveillance data collected by Medicare. The drug plan’s call centers received weekly survey phone calls to track the length of time to reach a live customer service representative. The drug plan’s “Customer Service for Current Members – Part D” phone number was monitored.
  
  
• Calls Disconnected When Customer Calls Drug Plan: The data used to measure the performance of the drug plan’s customer service come from call center surveillance data collected by Medicare. The drug plan’s call centers received weekly survey phone calls to track the number of times a call was disconnected by the drug plan. The drug plan’s “Customer Service for Current Members – Part D” phone number was monitored.
  
  
• Time on Hold When Pharmacist Calls Drug Plan: The data used to measure the performance of the drug plan’s customer service come from call center surveillance data collected by Medicare. The drug plan’s call centers received weekly survey phone calls to track the length of time to reach a live customer service representative. The drug plan’s “Pharmacy Technical Help Desk” phone number was monitored.
  
  
• Calls Disconnected When Pharmacist Calls Drug Plan: The data used to measure the performance of the drug plan’s customer service come from call center surveillance data collected by Medicare. The drug plan’s call centers received weekly survey phone calls to track the number of times a call was disconnected by the drug plan. The drug plan’s “Pharmacy Technical Help Desk” phone number was monitored.
  
  
• Complaints About the Drug Plan: The data used to find out the rate of complaints Medicare received about drug plans come from Medicare’s Complaints Tracking Module (CTM). These rates represent the number of complaints received for every 1,000 people enrolled in the drug plan (based on Medicare enrollment records), pro-rated to a 30-day basis.
  
  
• How Helpful is Your Plan When You Need Information: These data measure the performance of drug plans member satisfaction related to getting help from the drug plan. The data used for this measure come from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey.
  
  
• Rating of Drug Plan: These data measure the performance of drug plans member satisfaction related to the overall rating of the drug plan. The data used for this measure come from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey.

Drug Pricing and Patient Safety: This rating shows how the drug plan performs in the following drug price and safety areas:

• Whether the drug plan has complete information on members who need Extra Help.
  
  
• Whether the drug plan provides accurate drug price information for Medicare's Plan Finder Website and keeps drug prices stable during the year.
  
  
• The percent of the drug plan members 65 and older that use certain drugs with a high risk of side effects, when there may be safer drug choices.
  
  
• Whether drug plan members with diabetes who also have high blood pressure are given a type of blood pressure medication that is recommended for people with diabetes.

Using Your Plan To Get Your Prescriptions Filled:  This category shows how well drug plans make prescription drugs available to their members.

• Getting Prescriptions Easily: These data measure the performance of drug plans member satisfaction related the members’ ability to get prescriptions filled easily. The data used for this measure come from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey.
  
  
• Pharmacists Have Up-to-date Plan Enrollment Information: These data measure the performance of drug plans in sharing information important for filling prescriptions. The percentage includes the total number of enrollees who have complete records provided by the drug plan out of all drug plan enrollees in Medicare’s enrollment records. The data used for this measure come directly from Medicare’s Management Information Integrated Repository (MIIR).
  
  
• Pharmacists Have Up-to-date Information on Plan Members Who Need Extra Help: These data measure the performance of drug plans in sharing information important for filling prescriptions, especially for drug plan members who need extra help (also known as low income subsidy or LIS beneficiaries). The percentage includes the total number of LIS enrollees who have complete records provided by the drug plan out of all drug plan LIS enrollees. The data used for this measure come directly from Medicare enrollment records.
  
  
• Complaints about the Plan's Benefits and Access to Prescription Drugs: The data used to find out the rate of complaints Medicare received about drug plans come from Medicare’s Complaints Tracking Module (CTM). These rates represent the number of complaints received for every 1,000 people enrolled in the drug plan (based on Medicare enrollment records), pro-rated to a 30-day basis.
  
  
• Complaints about Joining and Leaving the Plan: The data used to find out the rate of complaints Medicare received about drug plans come from Medicare’s Complaints Tracking Module (CTM). These rates represent the number of complaints received for every 1,000 people enrolled in the drug plan (based on Medicare enrollment records), pro-rated to a 30-day basis.
  
  
• Delays in Appeals Decisions: The data used to find out the performance of drug plans in handling appeals come from a third party reviewer (also known as the Independent Review Entity or IRE). This organization evaluates Part D appeals after the drug plan’s review. A rate of IRE cases reviewed, because the drug plans did not make a timely decision, is calculated per 10,000 people enrolled in the drug plan (based on Medicare enrollment records).
  
  
• Reviewing Appeals Decisions: The data used to find out the performance of drug plans in handling appeals come from a third party reviewer (also known as the Independent Review Entity or IRE). This organization evaluates Part D appeals after the drug plan’s review. A percentage of cases in which the IRE agreed with the drug plans’ decision are displayed.  “No Appeals Review Required” means that the appeals for these drug plans did not reach the IRE for review.

Does the Drug Plan Make it Easy to Get Your Prescriptions Filled?  This category includes measures of how drug plans rate on the following areas:

• Plan members’ ability to get prescriptions filled easily.
  
  
•  Enrollment information the drug plan shares with pharmacists when a member needs a prescription filled.
  
  
• Information Medicare and the drug plan have about plan members who qualify for extra help.
  
  
• Complaints Medicare has received about the drug plan's benefits and access to prescription drugs. These complaints include situations where your drug plan membership card doesn’t work at the pharmacy, or a pharmacy is listed incorrectly on Medicare's website.
  
  
• Complaints Medicare has received about joining and leaving the drug plan. These complaints include situations where you don’t receive enrollment materials (like your membership card) from your drug plan, or you have difficulty switching to a new drug plan.
  
  
• How often a plan failed to make a timely appeals decision. Appeals are special kinds of requests you file when you disagree with certain decisions made by the drug plan about getting a prescription filled.

Why is this information important?
When drug plans do a good job of providing timely and appropriate access to your prescription drugs, you have a better chance of getting prescriptions when you need them and paying the right amount for them.

Drug Pricing Information:  This category shows how well drug plans are doing with pricing prescriptions and providing information on the Medicare website.

• Availability of Drug Coverage and Cost Information: The data used for this measure come from price files that drug plans submit for display on Medicare’s website (also known as Medicare’s Prescription Drug Plan Finder).
  
  
• How Often the Plan's Drug Prices Change: The data used for this measure come from price files that drug plans submit for display on Medicare’s website (also known as Medicare’s Prescription Drug Plan Finder).
  
  
• Complaints about the Plan's Drug Pricing and Out-of-pocket Costs: The data used to find out the rate of complaints Medicare received about drug plans come from Medicare’s Complaints Tracking Module (CTM). These rates represent the number of complaints received for every 1,000 people enrolled in the drug plan (based on Medicare enrollment records), pro-rated to a 30-day basis.

As a note, we deal on a regular basis with large amounts of data and find that we are naturally a bit skeptical about conclusions drawn from such information.  For instance, the CMS Ratings data represents a sample in a limited window of time (and the sample is often biased to the negative - just by human nature).

From our own experiences, we hear a lot of feedback about the Part D plans and were surprised to see that some companies that people complain about  had great Medicare ratings, while other companies that we know provide excellent service received lower rankings.

Perhaps the bottom line is that you should consider Medicare's Part D plan rating system, but use it only as additional guidance, not as a the deciding factor when seeking a new Part D plan.